Skip to main content

Architecture

This topic mainly introduces the CloudCanal technical architecture, including four parts: product architecture, kernel architecture, disaster recovery solution, and hybrid cloud network solution.

Product Architecture​

cloudcanal product arch overview

  • Console

    • Centralized management and control services that exist as web service clusters.
    • Carry productized functions, including data source/machine/data task lifecycle management, disaster recovery scheduling, monitoring and alarming, and metadata management.

  • Sidecar

    • Deployed on specific data migration synchronization machines.
    • Include obtaining DataJob configurations that need to run, starting and stopping DataJob processes, collecting and reporting DataJob status, and performing DataJob health checks.

  • CloudCanal Core

    • Deployed on specific data migration synchronization machines.
    • Perform specific data migration, synchronization, validation, and revision tasks.

Core Architecture​

cloudcanal core arch

  • DataSource plug-in

    • It includes data read/write, metadata acquisition logic, and corresponding drivers for each database, message, data warehouse, and other data sources.
    • Each plug-in is isolated by the Java class loader mechanism, and only the corresponding data source plug-in is loaded when the DataJob runs.

  • Core

    • It includes kernel code skeleton, operation filtering, metadata mapping, DDL conversion, custom data processing, and other parts.

  • Support

    • Contains metadata, task configuration, position management, monitoring metrics, and logic for interaction with console.

Disaster Recovery​

  • Console disaster recovery

    • It is solved through clustered deployment, and the stateful part is solved by the metadata database.

  • DataJob level 1 disaster recovery

    • When the sidecar process exits, the machine is abnormal, and the network is isolated, the console performs proactive disaster recovery scheduling based on the lease period and the status of the sidecar link.

  • DataJob level 2 disaster recovery

    • The sidecar process is normal and the task process is not normal, and the sidecar ensures that the DataJob it is responsible for runs in accordance with the state specified by the control through health monitoring, and is kept alive or dead.

Multiple/Hybrid Cloud Network​

CloudCanal uses a variety of network security measures to ensure the security of user data and information in order to accommodate multi-tenant, distributed system deployments.

  • Single direction connection

    • Sidecar nodes backlink Console, sidecar nodes do not actively expose network information.

  • HTTPS protocol

    • The Sidecar node and the Console communicate using the HTTPS protocol to prevent information theft and tampering.

  • Data in private network

    • All data transfer occurs on the user's intranet, and no data leakage occurs. All CloudCanal actions against data sources occur in the user's network environment.

  • AccessKey SecurityKey authentication

    • With TCP persistent connections, each connection is authenticated by the user's unique AccessKey and SecurityKey.

  • Request validation

    • Every request from the sidecar is verified for resource attribution.

  • Operation auditing

    • The operations of the sidecar node requesting the console are audited and can be traced.