Skip to main content

Redis Bidirectional Synchronization

Overview​

This topic introduces how CloudCanal does Redis bidirectional synchronization and anti-loop, and the features include:

  • Redis single-node, primary-standby, and clusters are supported.
  • Supports anti-loop for data initialization.
  • Supports anti-loop command expire or never expire settings.

Key Points​

Anti-loop Command​

CloudCanal Redis bidirectional synchronization uses auxiliary command to determine the loop, when receiving normal command, calculate its hash code, build auxiliary command key, query whether auxiliary command exist, if it exists, it is a loop, filtering.

CloudCanal has optimized batch and multi-threading for auxiliary commands pair-to-end writing and source-side queries, effectively improving synchronization performance.

Anti-loop compatibility data synchronization between Redis clusters, single nodes, and any combination of primary/standby nodes.

Multi Redis Nodes Subscription​

In order to simplify DataJob configuration, CloudCanal adopts a single-task multi-Redis node subscription mode to achieve data migration and synchronization, which is more convenient and reliable.

Example​

Install CloudCanal​

Add DataSource​

  • In this example, two Redis clusters with 3 primary and 3 standby each are used.
  • Log in to the CloudCanal, DataSource > Add DataSource , add 2 Redis clusters.
  • Modify the description of the DataSource to prevent the identification of the wrong database when create DataJob.redis_loop_ds

Create Forward DataJob​

  • DataJob->Create DataJob

  • In bidirectional synchronization, the forward link generally refers to the DataJob where the source has data and the target has no data, which involves the initialization of data at the peer end.

  • In the first page, select the source and target DataSources , and click Next Step.

    redis_loop_create_1

  • In the second page

    • Select Incremental, then check Full Data option.
    • Graying out starts automatically to set parameters after the DataJob is created.
    • Click Next Step.

    redis_loop_create_2

  • In the third page,click Create DataJob.

  • Details > Functions > Modify Parameters

    • Check Source tab, set deCycle to true and set deCycleEventExpireSec to 1200 (sec,for expire anti-loop command)
    • Save and start the DataJob.

    redis_loop_create_3

  • Wait for the forward DataJob to initialize the data and synchronize normally

    It is not recommended to create a reverse DataJob immediately after the forward synchronization DataJob is created, which involves the problem that when the repl-backlog-size setting is insufficient, the reverse DataJob starts to force FULL SYNC, resulting in new data being overwritten by old data.

    redis_loop_forward_job

Create Reverse DataJob​

  • DataJob->Create DataJob

  • In the first page, select the source and target DataSources(reverse selection of Forward DataJob), and click Next Step.

    redis_loop_create_reverse_1

  • In the second page

    • Select Incremental, and DO NOT check Full Data option.
    • Graying out starts automatically to set parameters after the DataJob is created.
    • Click Next Step.

    redis_loop_create_reverse_2

  • In the third page,click Create DataJob.

  • Details > Functions > Modify Parameters

    • Check Source tab, set deCycle to true and set deCycleEventExpireSec to 1200 (sec,for expire anti-loop command)
    • Save and start the DataJob.

    redis_loop_create_reverse_3

  • The DataJobs run normally.

    redis_loop_reverse_job

Test​

  • The source DataSource changes data, and the forward DataJob monitoring changes, but the reverse DataJob does not (that is, there is no loop) redis_loop_forward_gen_data redis_loop_forward_gen_data_src redis_loop_forward_gen_data_dst

  • The target DataSource changes data, and the reverse DataJob monitoring changes, and the forward DataJob does not (that is, there is no loop) redis_loop_reverse_gen_data redis_loop_reverse_gen_data_src redis_loop_reverse_gen_data_dst

  • Wait for the both side anti-loop auxiliary commands to expire and check that the data is consistent.

    • Source

    redis_loop_src_final

    • Target

    redis_loop_dst_final

FAQ​

Current Legacy Issues​

  • The FULL SYNC control is not accurate enough for primary/standby switch or too old offset, and there is a problem that old data overwrites new data due to FULL SYNC.
  • Anti-loop command are currently limited: FULL DUMP, SET, HSET, DEL.

Summary​

This topic briefly introduces how to use CloudCanal to build bidirectional synchronization of Redis to help users achieve remote multi-activity and disaster recovery business goals.