Skip to main content

3.4.1.0

CloudCanal-3.4.1.0​

Release Date: April 03th, 2024, Version Number: 3.4.1.0

New Connection​

  • Add Oracle -> RocketMQ structure migration, full data, incremental.
  • Add TiDB -> ClickHouse structure migration, full data, and incremental, data verification.

New Features​

  • Support Kafka / RocketMQ DSG message format for the target.
  • Support Kafka OGG KV JSON message format for the source.
  • Support Kafka SASL / SCRAM authentication methods (for Alibaba Cloud/AWS managed Kafka).
  • Support MySQL -> Kafka migration and synchronization for multiple schemas.
  • Support MySQL -> RocketMQ migration and synchronization for multiple schemas.
  • Support DDL event alerts for MySQL, PolarDbMySQL, and RDS for MySQL source endpoints (alerts are not set by default).
  • Support LogMiner in Oracle source endpoints to parse only archived logs (new parameter: archiveLogOnlyMode).
  • Support specifying the number of archived logs to be analyzed in a single LogMiner analysis when LogMiner is in ADD_FILE mode (new parameter: oraIncrLmAddFileLimit).
  • Support printing the view records of redo or archive logs analyzed by LogMiner in Oracle source endpoints (new parameter: oraPrintViewInLogs).
  • Support setting the SCN range size for each LogMiner analysis of redo or archive logs in Oracle source (new parameter: logMiningScnStep).
  • Support incremental materialized view mode (mlog) in Oracle, enriching the options for incremental from Oracle source.
  • Support adding virtual columns when modifying a task subscription.
  • Support including virtual column information when creating similar tasks.
  • Support modifying subscriptions for Kafka source (except for Kafka -> ElasticSearch).
  • Support retaining or filtering excess columns in Kafka source messages (core parameter: autoSyncNewCreatedColumn).
  • Support viewing table-to-database mappings for Kafka / RocketMQ -> MySQL.
  • Support modifying the bufferMemory parameter when writing to Kafka target.
  • Support time zone conversion for MySQL source during full synchronization (parameter: srcTimeZone), maintaining consistency with incremental synchronization capabilities.
  • Support sorting validation tasks by loss, inconsistency, or filter.
  • Support automatic installation, upgrade, and rollback of Sidecar using SSH passwordless login.
  • Support batch precise matching when creating tasks and filtering tables.

Optimization​

  • Optimized CHECK_POS heartbeat mode for MySQL source endpoints to record all binlog events for delayed comparison.
  • Optimized LRem and HIncrBy commands for Redis <-> Redis interaction.
  • Optimized Kafka driver loading method (external loading) to address security vulnerabilities. Default versions include kafka-clients 3.4.1 and kafka-client 2.8.2 (compatible with older Kafka versions).
  • Optimized RocketMQ driver loading method (external loading).
  • Optimized Hana driver loading method (external loading).
  • Optimized Common-text driver version to address security vulnerabilities.
  • Optimized logging of data writing in StarRocks, Doris, and SelectDB target endpoints, adding label information for easier troubleshooting of slow data writing.
  • Optimized Debezium format for Kafka source endpoints to handle inconsistent numbers of "before" and "after" fields during parsing.
  • Optimized SSH connection method by using a connection pool to cache session connections, eliminating the need for multiple connection establishments.

Problem Fixing​

  • Fixed the issue where modifying soTimeout for StarRocks / Doris initialization links had no effect.
  • Fixed the problem where default database loss during Hana connection creation led to permission issues.
  • Fixed the issue where changing the internal network address caused Console alerts to become ineffective.
  • Fixed the issue with the type of all synchronization links for TiDB source.
  • Fixed the problem of incorrect validation of data in time virtual columns.
  • Fixed the issue of custom code package loss after modifying the subscription of a custom code task.
  • Fixed the error that occurred when testing the connection to a Redis sentinel instance without a password while the master instance had a password.
  • Fixed the problem of TTL loss during full initialization (Key, Hash, List, Set, ZSet) for Redis -> Redis.
  • Fixed the issue where Oracle source endpoints in mlog incremental mode did not clean up consumed data.
  • Fixed the issue of data loss due to inconsistent order of composite primary keys in Oracle source.
  • Fixed the inaccurate NeedAutoCreate issue for Oracle -> Oracle TableSpace (referring to an error in structure migration when the target schema exists).
  • Fixed the problem of data loss in Oracle target during data validation due to unhandled spaces in CHAR/NCHAR types.
  • Fixed the error that occurred when parsing CHAR/NCHAR lengths exceeding 4000 in Oracle source.
  • Fixed the issue where similar tasks could not be created for Oracle source.
  • Fixed the problem where modifying subscriptions in Oracle -> StarRocks did not display correctly.
  • Fixed the issue where incorrect Json node order in data validation resulted in inconsistent data.
  • Fixed the problem with PostgreSQL -> MySQL incremental synchronization with Json data types.
  • Fixed the issue where null strings in time types caused conversion failures in PostgreSQL target.
  • Fixed the issue where full and incremental consumption partitions differed for OceanBase -> Kafka with the same primary key messages.
  • Fixed the issue where continuous carrying of DML statements in messages occurred in MySQL -> Kafka due to the binlog_rows_query_log_events session parameter setting.