Understanding Application.yml

The behavior of the consumer is configured through the application.yml file. This file instructs the consumer on which brokers to read data from, which backend database to persist the data to, how many threads to run, and more.

Configuration Reference

You can remove any configurations that are not relevant to your use case from the application.yml. The sample below demonstrates all currently supported configuration options. Any field marked as optional can be left out. You can also replace the value of any field with an environment variable.

spring:
  kafka:
    consumer:
      group-id: # e.g. myabc-consumer-1; Enter your consumer group ID here. 
                # You can get this information from your Solifi representative.
                # We encourage using a new group ID to support correct E2E testing.
                # Using an existing group ID could cause no data, because data offsets 
                # could be already read and committed from another application.
    bootstrap-servers: # e.g. pkc-89hc.ap-southeast-2.aws.confluent.cloud:9092
                       # This is the address of your Solifi brokers. 
                       # You can get this information from your Solifi representative. 
    properties:
      schema:
        registry:
          url: # e.g. https://myabc-123.ap-southeast-2.aws.confluent.cloud
               # This is the address of your schema registry.
          basic.auth.user.info: API-KEY:API-SECRET 
            # e.g. AOI8hCTKKA:51m2VSl0yn3515ixYLJN7S0ykawEMo5qtm2E2
            # Credentials to connect to the schema registry.
      security.protocol: SASL_SSL
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="API-KEY" password="API-SECRET";
        # You must enter your username and password for Kafka here.
      #partition.assignment.strategy: # Optional. Will default to RangeAssignor if not provided.
        # Options: org.apache.kafka.clients.consumer.RoundRobinAssignor
        #          org.apache.kafka.clients.consumer.StickyAssignor
        #          org.apache.kafka.clients.consumer.RangeAssignor
  
  # Datasources control where the messages are stored in the backend.
  # At least one backend source must be specified.
  datasource:
    ## Microsoft SQL
    #url: <sql server names>
    #driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
    #username: <sql username>
    #password: <sql password>
    
    ## PostgreSQL
    #url: jdbc:postgresql://192.156.31.82:5432/solifidb
    #driver-class-name: org.postgresql.Driver
    #username: postgres
    #password: postgres
    
    ## MySQL
    #url: jdbc:mysql://192.168.29.82:3306/solifidb
    #driver-class-name: com.mysql.cj.jdbc.Driver
    #username: root
    #password: mysql    
    
    ## Oracle
    #url: jdbc:oracle:thin:@192.156.31.82:1521/XE
    #driver-class-name: oracle.jdbc.OracleDriver
    #username: sys as sysdba
    #password: Oracle123$    
    
    ## MariaDB
    #url: jdbc:mariadb://192.156.31.82:3304/solifidb
    #driver-class-name: org.mariadb.jdbc.Driver
    #username: root
    #password: mariadb123

logging: # Optional
  level:
    com.limepoint.solifi: DEBUG
    # The consumer by default uses the log level of INFO.
    # WARNING: DEBUG mode will log records in clear text which may contain sensitive data.
    # Use DEBUG mode only sparingly.

solifi:
  prefix: # Required. Client prefix for Solifi brokers.
          # e.g. uat.myabc.ILS.canonical
          # You would get this information from Solifi.
  
  suffix: _v2 # Optional. If specified, consumer will remove the suffix from topic names
              # when creating tables. e.g. if topic is addl_lessor_nf_v2, 
              # table will be created as addl_lessor_nf
  
  topics-regex: # Optional. If specified, consumer constructs the list of topics 
                # based on the regex. Defaults to '.*' which reads all topics 
                # that begin with the prefix value.
                # If topics-regex and topics are both specified, topics takes precedence.
                # Supports any regex pattern supported by Java.
  
  topic-table-mapping: # Optional. Matches topic names and replaces with specified value.
    - match: "uat.client.FMO.canonical.*-"
      replace: ""
  
  topics: # Optional. List of topics to consume from Solifi brokers.
          # Format: <topic_name>:<table_name>:<audit_table_name>
          # Only <topic_name> is required.
    - addl_lessor_nf
    - cs_master_nf
    - address_nf
  
  support-unicode: # Optional. Defaults to false. 
                   # If true, creates all columns with unicode support (nvarchar).
                   # WARNING: Unicode occupies twice the space as non-unicode.
  
  audit: # Optional
    enabled: true # Optional. Defaults to false. Creates audit tables when true.
    include-all: true # Default true. Creates audit tables for all topics.
    audit-suffix: _audit # Suffix for audit table names.
  
  concurrency: 5 # Optional. Number of listener threads per consumer.
                 # Defaults to server CPU count (usually 8 or 10).
                 # For best performance, set equal to topic partition count.
  
  default-db-schema-name: # Optional. Only for MSSQL. Defaults to 'dbo'.
  
  license:
    path: # Path of the license file provided by LimePoint.
    signature:
      path: # Path of the signature file provided by LimePoint.
  
  timezone: # Optional. TZ identifier in TZDB.
            # Refer: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
            # If not present, all date/time stored in UTC.
            # We recommend keeping all values in UTC.

server: # Optional. Port for monitoring and statistics.
  port: 8080

Key Configuration Sections

Kafka Configuration

Property	Required	Description
`spring.kafka.consumer.group-id`	Yes	Consumer group ID from Solifi
`spring.kafka.bootstrap-servers`	Yes	Solifi broker address
`schema.registry.url`	Yes	Schema registry URL
`schema.registry.basic.auth.user.info`	Yes	Schema registry credentials
`sasl.jaas.config`	Yes	Kafka authentication credentials

Solifi Configuration

Property	Required	Description
`solifi.prefix`	Yes	Client prefix for Solifi brokers
`solifi.topics` or `solifi.topics-regex`	No	Topics to consume (defaults to all)
`solifi.license.path`	Yes	Path to license file
`solifi.license.signature.path`	Yes	Path to signature file

Optional Features

Property	Default	Description
`solifi.audit.enabled`	false	Enable audit tables
`solifi.support-unicode`	false	Enable unicode support
`solifi.concurrency`	CPU count	Number of listener threads
`solifi.timezone`	UTC	Timezone for date/time values

Using Environment Variables

You can replace any configuration value with an environment variable:

spring:
  kafka:
    bootstrap-servers: ${SOLIFI_BOOTSTRAP_SERVERS}
  datasource:
    username: ${SOLIFI_DATABASE_USERNAME}
    password: ${SOLIFI_DATABASE_PASSWORD}

Important

You must export the variables before starting the consumer, otherwise they will default to empty values.

Next Steps

View sample configurations
Start deploying the consumer

Configuration Reference​

Key Configuration Sections​

Kafka Configuration​

Solifi Configuration​

Optional Features​

Using Environment Variables​

Next Steps​