Sample Configurations

This page provides sample application.yml configurations for common use cases.

Read All Topics

This sample reads all topics that begin with uat.client.ILS.canonical (prefix) from a cluster and stores them in a MS SQL database.

solifi:
  prefix: uat.client.ILS.canonical
  audit:
    enabled: true
  license:
    path: /license.license
    signature:
      path: /sign256.sign
spring:
  kafka:
    consumer:
      group-id: clientname-dev
    bootstrap-servers: bootstrap-server-hostname:9092
    properties:
      schema:
        registry:
          url: https://registry-server-hostname
          basic.auth.user.info: REGISTRY_USERNAME:REGISTRY_PASSWORD
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule   required username='BROKER_USERNAME'   password='BROKER_PASSWORD';
  datasource:
    url: jdbc:sqlserver://mssqldb-ac-uat-sink:1433;database=master;encrypt=true;trustServerCertificate=true;
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: admin
    password: password

Using Environment Variables

Instead of hard coding credential values, use environment variables. You can use whatever name you want for the environment variable.

solifi:
  prefix: uat.client.ILS.canonical
  audit:
    enabled: true
  license:
    path: /license.license
    signature:
      path: /sign256.sign
spring:
  kafka:
    consumer:
      group-id: clientname-dev
    bootstrap-servers: ${SOLIFI_BOOTSTRAP_SERVERS}
    properties:
      schema:
        registry:
          url: https://registry-server-hostname
          basic.auth.user.info: ${SOLIFI_REGISTRY_USERNAME}:${SOLIFI_REGISTRY_PASSWORD}
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username=${SOLIFI_BROKER_USERNAME}   password=${SOLIFI_BROKER_PASSWORD};
  datasource:
    url: jdbc:sqlserver://mssqldb-ac-uat-sink:1433;database=master;encrypt=true;trustServerCertificate=true;
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: ${SOLIFI_DATABASE_USERNAME}
    password: ${SOLIFI_DATABASE_PASSWORD}

Export Variables First

You must export the variables before you start the consumer else they will default to empty values. You can export them on the command line or if using docker in the compose.yml file or in K8s environment variables.

Read Topics Matching A Pattern

This sample reads all topics that begin with uat.client.ILS.canonical (prefix) and start with cs_ (topics-regex).

solifi:
  prefix: uat.client.ILS.canonical
  topics-regex: cs_.*
  audit:
    enabled: true
  license:
    path: /license.license
    signature:
      path: /sign256.sign
spring:
  kafka:
    consumer:
      group-id: clientname-dev
    bootstrap-servers: bootstrap-server-hostname:9092
    properties:
      schema:
        registry:
          url: https://registry-server-hostname
          basic.auth.user.info: REGISTRY_USERNAME:REGISTRY_PASSWORD
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule   required username='BROKER_USERNAME'   password='BROKER_PASSWORD';
  datasource:
    url: jdbc:sqlserver://mssqldb-ac-uat-sink:1433;database=master;encrypt=true;trustServerCertificate=true;
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: admin
    password: password

Read Explicitly Defined Topics

This sample reads only a specific set of topics.

solifi:
  prefix: uat.client.ILS.canonical
  topics:
    - cs_master_nf
    - as_addl_nf
  audit:
    enabled: true
  license:
    path: /license.license
    signature:
      path: /sign256.sign
spring:
  kafka:
    consumer:
      group-id: clientname-dev
    bootstrap-servers: bootstrap-server-hostname:9092
    properties:
      schema:
        registry:
          url: https://registry-server-hostname
          basic.auth.user.info: REGISTRY_USERNAME:REGISTRY_PASSWORD
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule   required username='BROKER_USERNAME'   password='BROKER_PASSWORD';
  datasource:
    url: jdbc:sqlserver://mssqldb-ac-uat-sink:1433;database=master;encrypt=true;trustServerCertificate=true;
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: admin
    password: password

Custom Schema in MSSQL

This sample stores all tables under a custom schema (ils) instead of the default dbo. This assumes you have created the custom database, the relevant users and schemas upfront.

solifi:
  prefix: uat.client.ILS.canonical
  default-db-schema-name: ils
  audit:
    enabled: true
  license:
    path: /license.license
    signature:
      path: /sign256.sign
spring:
  kafka:
    consumer:
      group-id: clientname-dev
    bootstrap-servers: bootstrap-server-hostname:9092
    properties:
      schema:
        registry:
          url: https://registry-server-hostname
          basic.auth.user.info: REGISTRY_USERNAME:REGISTRY_PASSWORD
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule   required username='BROKER_USERNAME'   password='BROKER_PASSWORD';
  datasource:
    url: jdbc:sqlserver://mssqldb-ac-uat-sink:1433;database=ils;encrypt=true;trustServerCertificate=true;
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: admin
    password: password

Unicode Support in MSSQL

This sample stores all columns with unicode support (nvarchar). Notice the property support-unicode.

solifi:
  prefix: uat.client.ILS.canonical
  default-db-schema-name: ils
  support-unicode: true
  audit:
    enabled: true
  license:
    path: /license.license
    signature:
      path: /sign256.sign
spring:
  kafka:
    consumer:
      group-id: clientname-dev
    bootstrap-servers: bootstrap-server-hostname:9092
    properties:
      schema:
        registry:
          url: https://registry-server-hostname
          basic.auth.user.info: REGISTRY_USERNAME:REGISTRY_PASSWORD
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule   required username='BROKER_USERNAME'   password='BROKER_PASSWORD';
  datasource:
    url: jdbc:sqlserver://mssqldb-ac-uat-sink:1433;database=ils;encrypt=true;trustServerCertificate=true;
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: admin
    password: password

Storage Considerations

Setting support-unicode: true will reduce the size of the columns the consumer creates because unicode occupies twice the space as non-unicode.

Regex Topic to Table Name Mapping

This sample uses topic-table-mapping to transform topic names into table names. For example, if your topics are named uat.client.FMO.canonical.calms-app_quote or uat.client.FMO.canonical.catalog-catalog_model, by default the tables would be named calms-app_quote and catalog-catalog_model.

Using topic-table-mapping, you can remove the prefixes like calms- or catalog-:

solifi:
  prefix: uat.client.FMO.canonical
  topic-table-mapping:
    - match: "uat.client.FMO.canonical.*-"
      replace: ""
  audit:
    enabled: true
  license:
    path: /license.license
    signature:
      path: /sign256.sign
spring:
  kafka:
    consumer:
      group-id: clientname-dev
    bootstrap-servers: bootstrap-server-hostname:9092
    properties:
      schema:
        registry:
          url: https://registry-server-hostname
          basic.auth.user.info: REGISTRY_USERNAME:REGISTRY_PASSWORD
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule   required username='BROKER_USERNAME'   password='BROKER_PASSWORD';
  datasource:
    url: jdbc:sqlserver://mssqldb-ac-uat-sink:1433;database=ils;encrypt=true;trustServerCertificate=true;
    driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: admin
    password: password

About topic-table-mapping

The topic-table-mapping property is only used for mapping topic names to table names. It does not direct the consumer or control how many topics are fetched from the broker. The match property takes any valid Java regex. The replace property can be any valid string.

Next Steps

Choose your deployment method
Learn about health monitoring

Read All Topics​

Using Environment Variables​

Read Topics Matching A Pattern​

Read Explicitly Defined Topics​

Custom Schema in MSSQL​

Unicode Support in MSSQL​

Regex Topic to Table Name Mapping​

Next Steps​