Skip to main content

Change Log

This page documents all releases of the LimePoint Solifi Consumer.

Release 2.2.4​

Bug Fixes​

  • Fixed a bug introduced in 2.2.0 which ignored the topic-table-mapping configuration that resulted in tables being created incorrectly.
  • Added exception reporting if the license file is corrupted.

Enhancements​

  • Added more logging to indicate the flow between the consumer and upstream brokers (e.g., consumers now log messages starting with Sending acknowledgement once processing is complete).
  • Updated the default max-poll-records to 5000 and pollTimeOut to 5 seconds.
  • Updated initial-load-mode to be a two-stage process to improve memory footprint. The consumer no longer requires excessive memory to run in initial-load-mode. See Initial Load Mode for details.
  • Added a new property save-full-audit which defaults to false. This property is applicable only during initial-load-mode and forces the consumer to save all audit events in the topics. See Initial Load Mode for details.

Upgrade Notes​

Auditing Users

If any of your existing audit tables have a primary key that does not include all of the following columns, you will need to perform a full data resynchronization:

  • lp_kafka_partition
  • lp_kafka_offset
  • lp_db_insert_user
  • lp_db_insert_date

If you are unable to perform a full resync, please contact LimePoint Support for guidance on correcting the primary keys without dropping tables.

Release 2.2.1​

Bug Fixes​

  • Fixed an issue where the consumer ignored the topic property when running in initial_load mode.

Release 2.2.0​

Enhancements​

  • Introduced a new consumer mode called initial_load. The consumer now supports two modes: streaming and initial_load.
    • By default, the consumer runs in streaming mode (same behavior as previous versions).
    • initial_load mode is recommended for customers handling large data loads (typically over 10 million messages).
    • See Initial Load Mode for details.
  • Updated error handling to clean up messages left behind in the MDC via other threads.

Release 2.1.1​

Enhancements​

  • Updated the datetime precision for the lp_db_insert_date column for all audit tables to precision 6 (helps with MS SQL mirroring issues).
    • Note: Only applicable for new audit tables; does not change existing tables.
  • Improved logging: consumer now appends partition:offset in front of every message.
  • Added msal4j:1.22.0 to help authenticate with MS Azure Identity.

Release 2.1.0​

Enhancements​

  • Platform upgrades including container images, JDK, database drivers, and Spring Boot.
  • Improved error logging with better messaging for database, network, or business exceptions.
  • Added insert_date to the audit logging tables.
  • Updated MSSQL driver support to not hold locks when using merge.

Release 2.0.8​

Enhancements​

  • Added support for unicode characters. By default, the consumer stores data without unicode. To enable unicode, set support-unicode: true in application.yml.
    • Note: Enabling unicode reduces column sizes as unicode occupies twice the space.
    • See Sample Configuration for details.
  • Updated the auditing function to add the consumer group id in audit tables. This enables customers to keep audit log history during data refreshes.
  • Added logging for delete events. The consumer now logs Received 'Delete' ConsumerRecord when it finds a record to be deleted.

Release 2.0.7​

Bug Fixes​

  • Fixed an issue with date columns not being recognized correctly if the schema has long type marked against the column.
    • This only affects clients who have not yet migrated to the v2 version of topics.
    • You may notice Updating Optional DATE schema from 'long' to 'int' for <column_name> messages in logs (can be safely ignored).

Release 2.0.6​

Enhancements​

  • Added support for monitoring via Prometheus endpoint. The consumer now exposes a /actuator/prometheus endpoint.
  • See Health Monitoring for details.

Release 2.0.5​

Enhancements​

  • Added support for using Regex to manage table-to-topic mappings via the solifi.topic-table-mapping property.
  • This enables clients to remove prefixes in topic names when creating tables.
  • See Sample Configuration for details.

Release 2.0.4​

Enhancements​

  • Added support for overriding the default schema used when creating tables in MSSQL server.
  • The property solifi.default-db-schema-name defaults to dbo when not specified.
  • See Sample Configuration for details.

Release 2.0.3​

Enhancements​

  • Introduced a new property topics-regex which uses regex pattern matching to automatically look up topics.
  • Default value is .* (reads all available topics).
  • The consumer periodically checks (every 5 mins) for new topics.
  • With this feature, you no longer need to restart the consumer when adding new topics.
  • If topics-regex and topics are both specified, topics takes precedence.

Release 2.0.2​

Changes​

  • Updated auditing tables to add a prefix of lp_ for metadata columns to avoid conflicts with database column names.

Release 2.0.1​

Architecture Changes​

Version 1 of the consumer relied on SQL files to manage database tables. This approach had limitations as any schema changes in the upstream Solifi broker required a new consumer release.

From version 2.x onward, the architecture was restructured:

  • ❌ No longer ships SQL files
  • ✅ Detects required schemas from the Schema Registry
  • ✅ Dynamically creates or updates tables in the backend database
First Run Behavior

When running the consumer for the first time, you might see errors in the logs related to table creation. These can be safely ignored as they are caused by multiple threads attempting to create the same table simultaneously.

Getting the Latest Version​

Contact LimePoint Support to obtain the latest version of the consumer.