Skip to main content

Split Synchronizer

By default, FME's SDKs keep segment and feature flag data synchronized as users navigate across disparate systems, treatments, and conditions. However, some languages, do not have a native capability to keep a shared local cache of this data to properly serve treatments. For these cases, we built the Split Synchronizer service.

This tool coordinates the sending and receiving of data to a remote datastore that all of your processes can share to pull data for the evaluation of treatments. Out of the box, FME supports Redis as a remote datastore, and so the Split Synchronizer uses Redis as the cache for your SDKs when evaluating treatments. It also posts impression and event data and metrics generated by the SDKs back to Harness servers, for exposure in the user interface or sending to the data integration of your choice. The Synchronizer service runs as a standalone process in dedicated or shared servers and it does not affect the performance of your code, or FME's SDKs.

Split Synchronizer version 5.0 available!

Since version 5.0.0 of the split-synchronizer, there's only one operation mode. What was once proxy mode is now a separate tool called Split proxy. This version includes a more performant way to evict impressions & events from redis that allows customers to handle much greater volumes of data, while protecting your flags from being evicted if the redis instance runs low on memory.

Currently, the SDKs supported by this versions are: Ruby 6.0.0 PHP 6.0.0 Node.js 10.6.0 Go 3.0.0 Python 7.0.0 .NET 4.0.0 Java 4.4.0

warning

If you are upgrading from Split Synchronizer version 4.x or below to 5.x, some of the configuration and environment parameter names have been changed. Refer to the Configuration section and modify the parameters names accordingly.

Supported SDKs​

The Split Synchronizer works with most of the languages that FME supports.

Synchronizer Compatibility Matrix​

LanguageSupported Synchronizer VersionsSupported SDK Versions
JavaNot implemented—
JavaScript1.x, 2.xJavaScript SDK 9.x, 10.x
Ruby1.6, 1.7, 1.8, 2.xRuby SDK 4.x, 5.x, 6.x (required for 2.x features)
PHP1.6, 1.7, 1.8, 2.xPHP SDK 5.x
Python1.6, 1.7, 1.8, 2.xPython SDK 5.x
.NET/.NET Core1.6, 1.7, 1.8, 2.x.NET/Core 2.1+, 3.x
Go Lang1.x, 2.xGoLang SDK 1.x
iOSN/A—
AndroidN/A—

Notes:

  • “Not implemented” means no Synchronizer support for that language.
  • “N/A” means not applicable. The Synchronizer is not used for those platforms.
  • SDK versions noted are minimum or recommended versions tested with each Synchronizer version.

If you are looking for a language that is not listed here, contact the support team at support@split.io to discuss your options.

Overall Architecture​

The service performs five actions:

  • Fetch feature flags: Retrieve the feature flag definitions.
  • Fetch segments: Retrieve your set segments lists.
  • Post impressions: Send to Harness servers the generated impressions by the SDK.
  • Post telemetry: Send to Harness servers different metrics of the SDK, such as latencies.
  • Post events: Send to Harness servers the generated events by the SDK .track method.
Split-Sync v5.0.0 pipelined data eviction

Starting with split-sync v5.0.0, we've introduced a new approach to impressions and events eviction. This replaces the previous approach of periodically fetching & posting impressions and events. Our new approch feature flags this task in 3 parts, a thread dedicated to fetching data from redis and placing it in a buffer, N threads (where N is derived from the number of available CPU cores) dedicated to parsing, formatting the data and placing it in a second buffer, and N (configurable) threads that pick the data and post it to Harness servers. The result is a significant increase in throughput, that will better suit customers which operate on big volumes of data.

Architecture​

Setup​

The service is available via Docker or command line and its source code is available at https://github.com/splitio/split-synchronizer.

  • Pull the image: docker pull splitsoftware/split-synchronizer
  • Run as:
docker run --rm --name split-synchronizer \
 -p 3010:3010 \
 -e SPLIT_SYNC_APIKEY="your-sdk-key" \
 -e SPLIT_SYNC_REDIS_HOST=<your-redis-host> \
 -e SPLIT_SYNC_REDIS_PORT=<your_redis_port> \
 splitsoftware/split-synchronizer
Synchronizer mode with local redis instance

Sometimes, when building POCs or testing the synchronizer locally, you might want to launch our docker container image, pointing to a local redis server (or another container with redis, whose port has been mapped to a local one). In such case, you should consider adding the option --network="host" (appending it to the command shown above) when launching the synchronizer. This will allow you to use -e SPLIT_SYNC_REDIS_HOST="localhost", with the split-synchronizer container properly reaching your local redis server.

Docker configuration

The Advanced configuration section includes additional Docker information in the column Docker environment variable.

See Deploy Synchronizer Docker Container in AWS ECS for a full step-by-step AWS deployment guide.

Command line​

To install and run the service from command line, follow the steps below depending of your platform.

Linux​

On Linux systems, the Synchronizer service install script is invoked with this.

curl -L -o install_linux.bin 'https://downloads.split.io/synchronizer/install_split_sync_linux.bin' && chmod 755 install_linux.bin && ./install_linux.bin

OSX​

On OSX systems, the Synchronizer service install script is invoked with this.

curl -L -o install_osx.bin 'https://downloads.split.io/synchronizer/install_split_sync_osx.bin' && chmod 755 install_osx.bin && ./install_osx.bin

Windows​

On Microsoft Windows systems, follow these steps:

  1. Download the app from https://downloads.split.io/synchronizer/split_sync_windows.zip.

  2. Unzip the downloaded file.

  3. Run it!

Download previous versions

The links above point to the latest version. To download a previous version of split-sync, go to https://downloads.split.io/synchronizer/downloads.sync.html.

Run the service​

To run the service, paste the snippet below into your command line terminal and add in your SDK-Key.

Linux/Mac​

split-sync -apikey "your_sdk_key"

Windows​

Open the cmd terminal or the PowerShell terminal, go to (cd) unzipped Split Synchronizer folder, and type:

split-sync.exe -apikey "your_sdk_key"
Redis instance

On the samples above, Redis is running as a local service with default host: localhost and default port: 6379. For further information, see Advanced configuration.

Redis database

To maximize performance and isolation, we recommend connecting to a Redis database dedicated to the Split Synchronizer. For further information, see Advanced configuration.

Redis Sentinel support

Split Synchronizer also supports Redis Sentinel (v2) replication. For further information about Redis Sentinel, refer to the Sentinel Documentation.

Redis Cluster support

Split Synchronizer supports Redis Cluster with Redis^3.0.0. For further information about Redis Cluster, refer to the Cluster Documentation.

You can run the service with the simple steps above, but the system is more stable in your production environment when you run the job with a scheduling system. We recommend starting the synchronizer via supervisord, a daemon that launches other processes and ensures they stay running.

To use supervisord, make sure that it is installed on your machine. You can get help on the installation at the official Supervisord documentation.

When supervisord is installed into your project, copy and paste the program below anywhere into the supervisord.conf file that should now be in your project.

[program:splitio_sync]
command=/usr/local/bin/split-sync -config /path/to/your/config.file.json
process_name = SplitIO
numprocs = 1
autostart=true
autorestart=true
user = your_user
stderr_logfile=/var/log/splitio.err.log
stderr_logfile_maxbytes = 1MB
stdout_logfile=/var/log/splitio.out.log
stdout_logfile_maxbytes = 1MB

Advanced configuration​

The Synchronizer service has a number of knobs for configuring performance. Each knob is tuned to a reasonable default, however, you can override the default values by changing a splitio.config.json file or by setting your customer values as parameters of -config in the command line option. In this section, we lay out all the different knobs you can configure for performance, Redis, and logging.

The splitio.config.json file provided via the -config option lets you control how often the synchronizer fetches data from Harness servers. You can create a sample JSON file automatically with default values by running this command.

./split-sync -write-default-config "/home/someuser/splitio.config.json"
Configuration path file

Save the JSON config file on your server in your desired folder. For instance, on Linux systems, it could be saved in the etc folder. Remember to set the right path as the -config parameter.

{
  "apikey": "YOUR_SDK_KEY",
  "ipAddressEnabled": true,
  "initialization": {
    "timeoutMS": 10000,
    "forceFreshStartup": false
  },
  "storage": {
    "type": "redis",
    "redis": {
      "host": "localhost",
      "port": 6379,
      "db": 0,
      "username": "",
      "password": "",
      "prefix": "",
      "network": "tcp",
      "maxRetries": 0,
      "dialTimeout": 5,
      "readTimeout": 10,
      "writeTimeout": 5,
      "poolSize": 10,
      "sentinelReplication": false,
      "sentinelAddresses": "",
      "sentinelMaster": "",
      "clusterMode": false,
      "clusterNodes": "",
      "keyHashTag": "",
      "enableTLS": false,
      "tlsServerName": "",
      "caCertificates": null,
      "tlsSkipNameValidation": false,
      "tlsClientCertificate": "",
      "tlsClientKey": ""
    }
  },
  "sync": {
    "splitRefreshRateMs": 60000,
    "segmentRefreshRateMs": 60000,
    "impressionsMode": "optimized",
    "advanced": {
      "streamingEnabled": true,
      "httpTimeoutMs": 30000,
      "internalTelemetryRateMs": 3600000,
      "telemetryPushRateMs": 60000,
      "impressionsFetchSize": 0,
      "impressionsProcessConcurrency": 0,
      "impressionsProcessBatchSize": 0,
      "impressionsPostConcurrency": 0,
      "impressionsPostSize": 0,
      "impressionsAccumWaitMs": 0,
      "eventsFetchSize": 0,
      "eventsProcessConcurrency": 0,
      "eventsProcessBatchSize": 0,
      "eventsPostConcurrency": 0,
      "eventsPostSize": 0,
      "eventsAccumWaitMs": 0
    }
  },
  "admin": {
    "host": "0.0.0.0",
    "port": 3010,
    "username": "",
    "password": "",
    "secureChecks": false
  },
  "integrations": {
    "impressionListener": {
      "endpoint": "",
      "queueSize": 100
    },
    "slack": {
      "webhook": "",
      "channel": ""
    }
  },
  "logging": {
    "level": "info",
    "output": "stdout",
    "rotationMaxFiles": 10,
    "rotationMaxSizeKb": 0
  },
  "healthcheck": {
    "app": {
      "storageCheckRateMs": 3600000
    }
  }
}
Command line parameters

All the options available in the JSON file are also included as command line options. Run the command followed by the -help option to see more details, or just keep reading this documentation page.

Methods to configure the Split Synchronizer​

You can configure the Split Synchronizer service using the command line or by directly editing the above mentioned JSON configuration file.

Config values priority

All config values are set with a default value that you can see in the example JSON file above. You can overwrite the default value from the JSON config file, and you can overwrite the JSON config file from the command line. See a sample below for how to do that via command line.

split-sync -config "/etc/splitio.config.json" -log-level "debug" -redis-pass "somePass"

Handling High Impression Rates with Synchronizer​

When using a server-side SDK with the Split Synchronizer and Redis, it is important to properly configure the Synchronizer to handle a high load of incoming impressions efficiently.

Key Configuration Parameters​

The Split Synchronizer (version 1.6.0 and above) exposes several parameters that control impression processing performance:

  • impressionsMaxSize: Maximum size for the impressions queue.
  • impressionsRefreshRate: How often impressions are processed and sent (in seconds).
  • impressionsThreads: Number of threads handling impressions processing.
  • impressionsPerPost: Number of impressions sent per post request.

Adjust these parameters according to your expected impression volume. Because the Synchronizer uses multithreading, increasing impressionsThreads can reduce latency in posting impressions.

Below is an example JSON configuration designed to handle approximately 100,000 impressions per minute. Update the API Key, Redis host, port, and database number to match your environment before applying.

{
  "apiKey": "YOUR_API_KEY",
  "proxy": {
    "port": 3000,
    "adminPort": 3010,
    "adminUsername": "",
    "adminPassword": "",
    "dashboardTitle": "",
    "persistInFilePath": "",
    "impressionsMaxSize": 10485760,
    "eventsMaxSize": 10485760,
    "auth": {
      "sdkAPIKeys": [
        "SDK_API_KEY"
      ]
    }
  },
  "redis": {
    "host": "localhost",
    "port": 6379,
    "db": 0,
    "password": "",
    "prefix": "",
    "network": "tcp",
    "maxRetries": 0,
    "dialTimeout": 5,
    "readTimeout": 10,
    "writeTimeout": 5,
    "poolSize": 10,
    "sentinelReplication": false,
    "sentinelAddresses": "",
    "sentinelMaster": ""
  },
  "sync": {
    "admin": {
      "adminPort": 3010,
      "adminUsername": "",
      "adminPassword": "",
      "dashboardTitle": ""
    }
  },
  "log": {
    "verbose": false,
    "debug": false,
    "stdout": false,
    "file": "/tmp/split-agent.log",
    "fileMaxSizeBytes": 2000000,
    "fileBackupCount": 3,
    "slackChannel": "",
    "slackWebhookURL": ""
  },
  "impressionListener": {
    "endpoint": ""
  },
  "splitsRefreshRate": 60,
  "segmentsRefreshRate": 60,
  "impressionsRefreshRate": 20,
  "impressionsPerPost": 10000,
  "impressionsThreads": 5,
  "eventsPushRate": 60,
  "eventsConsumerReadSize": 10000,
  "eventsConsumerThreads": 1,
  "metricsRefreshRate": 60,
  "httpTimeout": 60
}

CLI Configuration options and its equivalents in JSON & Environment variables​

Split Synchronizer version 5.0 boolean options change

In order to reduce the issues because of typos and confusion due to "multiple words & case meaning the same", since version 5.0.0 of the split-synchronizer, the only accepted values for boolean flags are "true" & "false" in lowercase. Things like "enabled", "on", "yes", "tRue" will result in an error at startup. This applies to JSON, CLI arguments & environment variables.

Command line optionJSON optionEnvironment variable (container-only)Description
log-levellevelSPLIT_SYNC_LOG_LEVELLog level (error|warning|info|debug|verbose)
log-outputoutputSPLIT_SYNC_LOG_OUTPUTWhere to output logs (defaults to stdout)
log-rotation-max-filesrotationMaxFilesSPLIT_SYNC_LOG_ROTATION_MAX_FILESMax number of files to keep when rotating logs
log-rotation-max-size-kbrotationMaxSizeKbSPLIT_SYNC_LOG_ROTATION_MAX_SIZE_KBMax file size before rotating log files.
admin-hosthostSPLIT_SYNC_ADMIN_HOSTHost where the admin server will listen
admin-portportSPLIT_SYNC_ADMIN_PORTAdmin port where incoming connections will be accepted
admin-usernameusernameSPLIT_SYNC_ADMIN_USERNAMEHTTP basic auth username for admin endpoints
admin-passwordpasswordSPLIT_SYNC_ADMIN_PASSWORDHTTP basic auth password for admin endpoints
admin-secure-hcsecureChecksSPLIT_SYNC_ADMIN_SECURE_HCSecure Healthcheck endpoints as well.
admin-tls-enabledenabledSPLIT_SYNC_ADMIN_TLS_ENABLEDEnable HTTPS on proxy endpoints.
admin-tls-client-validationclientValidationSPLIT_SYNC_ADMIN_TLS_CLIENT_VALIDATIONEnable client cert validation.
admin-tls-server-nameserverNameSPLIT_SYNC_ADMIN_TLS_SERVER_NAMEServer name as it appears in provided server-cert.
admin-tls-cert-chain-fncertChainFnSPLIT_SYNC_ADMIN_TLS_CERT_CHAIN_FNX509 Server certificate chain.
admin-tls-private-key-fnprivateKeyFnSPLIT_SYNC_ADMIN_TLS_PRIVATE_KEY_FNPEM Private key file name.
admin-tls-client-validation-root-certclientValidationRootCertFnSPLIT_SYNC_ADMIN_TLS_CLIENT_VALIDATION_ROOT_CERTX509 root cert for client validation.
admin-tls-min-tls-versionminTlsVersionSPLIT_SYNC_ADMIN_TLS_MIN_TLS_VERSIONMinimum TLS version to allow X.Y.
admin-tls-allowed-cipher-suitesallowedCipherSuitesSPLIT_SYNC_ADMIN_TLS_ALLOWED_CIPHER_SUITESComma-separated list of cipher suites to allow.
impression-listener-endpointendpointSPLIT_SYNC_IMPRESSION_LISTENER_ENDPOINTHTTP endpoint to forward impressions to
impression-listener-queue-sizequeueSizeSPLIT_SYNC_IMPRESSION_LISTENER_QUEUE_SIZEmax number of impressions bulks to queue
slack-webhookwebhookSPLIT_SYNC_SLACK_WEBHOOKslack webhook to post log messages
slack-channelchannelSPLIT_SYNC_SLACK_CHANNELslack channel to post log messages
apikeyapikeySPLIT_SYNC_APIKEYSplit Server-side SDK api-key
ip-address-enabledipAddressEnabledSPLIT_SYNC_IP_ADDRESS_ENABLEDBundle host's ip address when sending data to Harness FME
timeout-mstimeoutMSSPLIT_SYNC_TIMEOUT_MSHow long to wait until the synchronizer is ready
snapshotsnapshotSPLIT_SYNC_SNAPSHOTSnapshot file to use as a starting point
force-fresh-startupforceFreshStartupSPLIT_SYNC_FORCE_FRESH_STARTUPWipe storage before starting the synchronizer
storage-typetypeSPLIT_SYNC_STORAGE_TYPEStorage driver to use for caching feature flags and segments and user-generated data
split-refresh-rate-mssplitRefreshRateMsSPLIT_SYNC_SPLIT_REFRESH_RATE_MSHow often to refresh feature flags
segment-refresh-rate-mssegmentRefreshRateMsSPLIT_SYNC_SEGMENT_REFRESH_RATE_MSHow often to refresh segments
impressions-modeimpressionsModeSPLIT_SYNC_IMPRESSIONS_MODEwhether to send all impressions for debugging
streaming-enabledstreamingEnabledSPLIT_SYNC_STREAMING_ENABLEDEnable/disable streaming functionality
http-timeout-mshttpTimeoutMsSPLIT_SYNC_HTTP_TIMEOUT_MSTotal http request timeout
internal-metrics-rate-msinternalTelemetryRateMsSPLIT_SYNC_INTERNAL_METRICS_RATE_MSHow often to send internal metrics
telemetry-push-rate-mstelemetryPushRateMsSPLIT_SYNC_TELEMETRY_PUSH_RATE_MShow often to flush sdk telemetry
impressions-fetch-sizeimpressionsFetchSizeSPLIT_SYNC_IMPRESSIONS_FETCH_SIZEImpression fetch bulk size
impressions-process-concurrencyimpressionsProcessConcurrencySPLIT_SYNC_IMPRESSIONS_PROCESS_CONCURRENCY#Threads for processing imps
impressions-process-batch-sizeimpressionsProcessBatchSizeSPLIT_SYNC_IMPRESSIONS_PROCESS_BATCH_SIZESize of imp processing batchs
impressions-post-concurrencyimpressionsPostConcurrencySPLIT_SYNC_IMPRESSIONS_POST_CONCURRENCY#concurrent imp post threads
impressions-post-sizeimpressionsPostSizeSPLIT_SYNC_IMPRESSIONS_POST_SIZEMax #impressions to send per POST
impressions-accum-wait-msimpressionsAccumWaitMsSPLIT_SYNC_IMPRESSIONS_ACCUM_WAIT_MSMax ms to wait to close an impressions bulk
events-fetch-sizeeventsFetchSizeSPLIT_SYNC_EVENTS_FETCH_SIZEHow many impressions to pop from storage at once
events-process-concurrencyeventsProcessConcurrencySPLIT_SYNC_EVENTS_PROCESS_CONCURRENCY#Threads for processing imps
events-process-batch-sizeeventsProcessBatchSizeSPLIT_SYNC_EVENTS_PROCESS_BATCH_SIZESize of imp processing batchs
events-post-concurrencyeventsPostConcurrencySPLIT_SYNC_EVENTS_POST_CONCURRENCY#concurrent imp post threads
events-post-sizeeventsPostSizeSPLIT_SYNC_EVENTS_POST_SIZEMax #impressions to send per POST
events-accum-wait-mseventsAccumWaitMsSPLIT_SYNC_EVENTS_ACCUM_WAIT_MSMax ms to wait to close an events bulk
redis-hosthostSPLIT_SYNC_REDIS_HOSTRedis server hostname
redis-portportSPLIT_SYNC_REDIS_PORTRedis Server port
redis-dbdbSPLIT_SYNC_REDIS_DBRedis DB
redis-passpasswordSPLIT_SYNC_REDIS_PASSRedis password
redis-userusernameSPLIT_SYNC_REDIS_USERRedis username
redis-prefixprefixSPLIT_SYNC_REDIS_PREFIXRedis key prefix
redis-networknetworkSPLIT_SYNC_REDIS_NETWORKRedis network protocol
redis-max-retriesmaxRetriesSPLIT_SYNC_REDIS_MAX_RETRIESRedis connection max retries
redis-dial-timeoutdialTimeoutSPLIT_SYNC_REDIS_DIAL_TIMEOUTRedis connection dial timeout
redis-read-timeoutreadTimeoutSPLIT_SYNC_REDIS_READ_TIMEOUTRedis connection read timeout
redis-write-timeoutwriteTimeoutSPLIT_SYNC_REDIS_WRITE_TIMEOUTRedis connection write timeout
redis-poolpoolSizeSPLIT_SYNC_REDIS_POOLRedis connection pool size
redis-sentinel-replicationsentinelReplicationSPLIT_SYNC_REDIS_SENTINEL_REPLICATIONRedis sentinel replication enabled.
redis-sentinel-addressessentinelAddressesSPLIT_SYNC_REDIS_SENTINEL_ADDRESSESList of redis sentinels
redis-sentinel-mastersentinelMasterSPLIT_SYNC_REDIS_SENTINEL_MASTERName of master
redis-cluster-modeclusterModeSPLIT_SYNC_REDIS_CLUSTER_MODERedis cluster enabled.
redis-cluster-nodesclusterNodesSPLIT_SYNC_REDIS_CLUSTER_NODESList of redis cluster nodes.
redis-cluster-key-hashtagkeyHashTagSPLIT_SYNC_REDIS_CLUSTER_KEY_HASHTAGkeyHashTag for redis cluster.
redis-tlsenableTLSSPLIT_SYNC_REDIS_TLSUse SSL/TLS for connecting to redis
redis-tls-server-nametlsServerNameSPLIT_SYNC_REDIS_TLS_SERVER_NAMEServer name to use when validating a server public key
redis-tls-ca-certscaCertificatesSPLIT_SYNC_REDIS_TLS_CA_CERTSRoot CA certificates to connect to a redis server via SSL/TLS
redis-tls-skip-name-validationtlsSkipNameValidationSPLIT_SYNC_REDIS_TLS_SKIP_NAME_VALIDATIONBlindly accept server's public key.
redis-tls-client-certificatetlsClientCertificateSPLIT_SYNC_REDIS_TLS_CLIENT_CERTIFICATEClient certificate signed by a known CA
redis-tls-client-keytlsClientKeySPLIT_SYNC_REDIS_TLS_CLIENT_KEYClient private key matching the certificate.
storage-check-rate-msstorageCheckRateMsSPLIT_SYNC_STORAGE_CHECK_RATE_MSHow often to check storage health
flag-sets-filterflagSetsFilterSPLIT_SYNC_FLAG_SETS_FILTERThis setting allows the Split Synchronizer to only synchronize the feature flags in the specified flag sets, avoiding unused or unwanted flags from being synced on the Split Synchronizer instance, bringing all the benefits from a reduced payload.

Deploying Synchronizer on Heroku​

Follow these steps to deploy the Synchronizer container on Heroku:

  1. Clone the GitHub repository or download the source code and unzip it to a new folder (e.g., mysync).

  2. Open a terminal and cd to that folder.

  3. Run this command to create a Heroku app:

    heroku create

  4. You should see a response like:

    Creating app... done, ⬢ secret-anchorage-16496
    https://secret-anchorage-16496.herokuapp.com/ | https://git.heroku.com/secret-anchorage-16496.git

  5. Set the stack to container and add the Go language buildpack:

    heroku stack:set container
    heroku buildpacks:set heroku/go

  6. Open the existing Dockerfile and replace the last line:

    ENTRYPOINT ["sh", "./entrypoint.sh"]

    with:

    CMD ["sh", "./entrypoint.sh"]

    Heroku only supports CMD for containers.

  7. Add the following content to a heroku.yml file:

    build:
     docker:
       web: Dockerfile
       worker:
         dockerfile: Dockerfile

  8. Add a Procfile with the content:

    worker: sh entrypoint.sh

  9. On the Heroku dashboard, open your app, go to the Settings tab, and add the following environment variables:

    • SPLIT_SYNC_API_KEY
    • SPLIT_SYNC_REDIS_HOST
    • SPLIT_SYNC_REDIS_PORT
    • SPLIT_SYNC_REDIS_DB
    • SPLIT_SYNC_REDIS_PASS

  10. Modify entrypoint.sh to map the admin dashboard to Heroku's $PORT environment variable.

    Replace the last line:

    exec split-sync ${PARAMETERS}

    with:

    exec split-sync ${PARAMETERS} -sync-admin-port $PORT

  11. Run these git commands to deploy to Heroku:

    git init
    git add .
    git commit -m "Deploy Synchronizer to Heroku"
    git push heroku master

  12. Check logs using heroku logs.

  13. You should see output similar to:

    2019-09-11T19:53:06.954882+00:00 app[worker.1]: __      ____        _ _ _
    2019-09-11T19:53:06.954887+00:00 app[worker.1]: / /__   / ___| _ __ | (_) |_
    2019-09-11T19:53:06.954890+00:00 app[worker.1]: / / \ \  \___ \| '_ \| | | __|
    2019-09-11T19:53:06.954891+00:00 app[worker.1]: \ \  \ \  ___) | |_) | | | |_
    2019-09-11T19:53:06.954893+00:00 app[worker.1]: \_\ / / |____/| .__/|_|_|\__|
    2019-09-11T19:53:06.954895+00:00 app[worker.1]: /_/        |_|
    2019-09-11T19:53:06.954897+00:00 app[worker.1]: 
    2019-09-11T19:53:06.954899+00:00 app[worker.1]: 
    2019-09-11T19:53:06.954922+00:00 app[worker.1]: 
    2019-09-11T19:53:06.954924+00:00 app[worker.1]: Split Synchronizer - Version: 2.5.1 (2178c61)
    2019-09-11T19:53:06.955154+00:00 app[worker.1]: Log file: /tmp/split-agent.log

  14. Access the Admin Dashboard by visiting https://[heroku-app-name].herokuapp.com/admin/dashboard.

Using the Synchronizer as a proxy service on Heroku​

Since Heroku maps only one port, you need to assign the proxy listener port to the $PORT environment variable.

  1. Update entrypoint.sh by replacing exec split-sync ${PARAMETERS} with exec split-sync ${PARAMETERS} -proxy-port $PORT.

  2. Modify Procfile by changing its content to web: sh entrypoint.sh.

  3. Add the following environment variables in the Heroku UI:

    • SPLIT_SYNC_PROXY (value: on)
    • SPLIT_SYNC_PROXY_SDK_APIKEYS (value: your customer API key)

  4. Deploy changes:

    git add .
    git commit -m "Enable proxy mode on Heroku"
    git push heroku master

  5. Configure SDKs. An example Java SDK configuration:

    SplitClientConfig config = SplitClientConfig.builder()
     .setBlockUntilReadyTimeout(5000)
     .endpoint("http://[heroku-app-name].herokuapp.com", "http://[heroku-app-name].herokuapp.com")
     .build();

    try {
        splitFactory = SplitFactoryBuilder.build("custom api key", config);
        client = splitFactory.client();
        client.blockUntilReady();
    } catch (Exception e) {
        System.out.print("Exception: " + e.getMessage());
    }

Since Heroku maps only a single port, the Admin Dashboard will not be accessible when running in proxy mode.

Listener​

The Split Synchronizer provides an impression listener that bulks post impressions to a user-defined HTTP endpoint.

The endpoint should expect a POST request, containing a JSON body with the following format.

JSON Impression
{
 "impressions": [
  {
   "testName": "feature1",
   "keyImpressions": [
    {
     "keyName": "user1",
     "treatment": "on",
     "time": 1502754901182,
     "changeNumber": -1,
     "label": ""
    },
    {
     "keyName": "user2",
     "treatment": "off",
     "time": 1502754876144,
     "changeNumber": -1,
     "label": ""
    }
   ]
  }
 ],
 "sdkVersion": "php-5.2.2",
 "machineIP": "208.63.222.7",
 "MachineName": ""
}

Currently, the configuration options are available in the integrations.impressionListener section of the JSON configuration file detailed in Advanced configuration.

Using a network proxy​

If you need to use a network proxy, configure proxies by setting the environment variables HTTP_PROXY and HTTPS_PROXY. The internal HTTP client reads those variables and uses them to perform the server request.

Example: Environment variables
$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"

Inject a certificate into a Synchronizer Docker image​

If the Synchronizer Docker container is running in a network that uses an SSL proxy, the Synchronizer may fail to authenticate the root certificate. This causes errors like the following when trying to connect to Harness FME servers to fetch feature flag definitions:

SPLITIO-AGENT | ERROR: 2020/08/19 14:42:51 fetchdataforproxy.go:209: Error fetching split changes  
Get https://sdk.split.io/api/splitChanges?since=-1: x509: certificate signed by unknown authority

To resolve this, you need to inject the root certificate into the Synchronizer Docker image by rebuilding it with the proxy certificates included.

  1. Clone the Synchronizer public repository:

    git clone https://github.com/splitio/split-synchronizer

  2. Change to the cloned directory:

    cd split-synchronizer

  3. Copy your proxy certificates (root, intermediate, and proxy certs) into this folder. For example:

    cp /path/to/certs/root.crt .
    cp /path/to/certs/intermediate.crt .
    cp /path/to/certs/proxy.pem .

  4. Open the Dockerfile in the split-synchronizer folder with a text editor, and just before the line containing EXPOSE 3000 3010, add:

    COPY root.crt /etc/ssl/certs/root.crt
    COPY intermediate.crt /etc/ssl/certs/intermediate.crt
    COPY proxy.pem /etc/ssl/certs/proxy.pem
    RUN cat /etc/ssl/certs/root.crt >> /etc/ssl/certs/ca-certificates.crt

  5. Save and close the Dockerfile.

  6. Build the new Docker image:

    docker build --tag split-sync:latest .

  7. Run the new image to verify it works (replace environment variables as needed). The http_proxy and https_proxy variables are optional based on your setup:

    docker run --rm --name split-sync -p 3010:3010 --net="host" \
     -e SPLIT_SYNC_API_KEY="SDK API KEY" \
     -e SPLIT_SYNC_LOG_STDOUT="on" \
     -e SPLIT_SYNC_LOG_DEBUG="true" \
     -e SPLIT_SYNC_LOG_VERBOSE="true" \
     -e SPLIT_SYNC_REDIS_HOST="Redis Host" \
     -e SPLIT_SYNC_REDIS_PORT=6379 \
     -e http_proxy="https://[internal proxy host]" \
     -e https_proxy="https://[internal proxy host]" \
     split-sync

Admin tools​

Endpoints​

The split-sync service has a set of endpoints and a dashboard to let DevOps and infra team monitor its status and cached data in real-time. By default the port is 3010 and for security reason supports HTTP Basic Authentication configured by the user.

/info/ping

A ping endpoint to monitor the service status. If the service is running, it sends the text response pong and the HTTP status code 200.

/info/version

Returns the split-sync version in JSON format.

{
  "version" : "1.1.0"
}

/info/uptime

Returns the uptime string representation in JSON format.

{
  "uptime" : "5d 3h 36m 39s"
}

/info/config Returns a JSON object describing the current configuration of the Synchronizer.

{
  "config": {
    "apikey": "*",
    "ipAddressEnabled": true,
    "initialization": {
      "timeoutMS": 10000,
      "forceFreshStartup": false
    },
    "storage": {
      "type": "redis",
      "redis": {
        "host": "localhost",
        "port": 6379,
        "db": 0,
        "username": "",
        "password": "",
        "prefix": "",
        "network": "tcp",
        "maxRetries": 0,
        "dialTimeout": 5,
        "readTimeout": 10,
        "writeTimeout": 5,
        "poolSize": 10,
        "sentinelReplication": false,
        "sentinelAddresses": "",
        "sentinelMaster": "",
        "clusterMode": false,
        "clusterNodes": "",
        "keyHashTag": "",
        "enableTLS": false,
        "tlsServerName": "",
        "caCertificates": null,
        "tlsSkipNameValidation": false,
        "tlsClientCertificate": "",
        "tlsClientKey": ""
      }
    },
    "sync": {
      "splitRefreshRateMs": 60000,
      "segmentRefreshRateMs": 60000,
      "impressionsMode": "optimized",
      "advanced": {
        "streamingEnabled": true,
        "httpTimeoutMs": 30000,
        "internalTelemetryRateMs": 3600000,
        "telemetryPushRateMs": 60000,
        "impressionsFetchSize": 0,
        "impressionsProcessConcurrency": 0,
        "impressionsProcessBatchSize": 0,
        "impressionsPostConcurrency": 0,
        "impressionsPostSize": 0,
        "impressionsAccumWaitMs": 0,
        "eventsFetchSize": 0,
        "eventsProcessConcurrency": 0,
        "eventsProcessBatchSize": 0,
        "eventsPostConcurrency": 0,
        "eventsPostSize": 0,
        "eventsAccumWaitMs": 0
      }
    },
    "admin": {
      "host": "0.0.0.0",
      "port": 3010,
      "username": "",
      "password": "",
      "secureChecks": false
    },
    "integrations": {
      "impressionListener": {
        "endpoint": "",
        "queueSize": 100
      },
      "slack": {
        "webhook": "",
        "channel": ""
      }
    },
    "logging": {
      "level": "info",
      "output": "stdout",
      "rotationMaxFiles": 10,
      "rotationMaxSizeKb": 0
    },
    "healthcheck": {
      "app": {
        "storageCheckRateMs": 3600000
      }
    }
  }
}

/health/application Returns a JSON object describing whether the synchronizer is healthy or not.

{
  "healthy": true,
  "healthySince": "2021-11-20T19:04:46.528242-03:00",
  "items": [
    {
      "name": "Splits",
      "healthy": true,
      "lastHit": "2021-11-20T19:04:49.079956-03:00"
    },
    {
      "name": "Segments",
      "healthy": true,
      "lastHit": "2021-11-20T19:04:49.268349-03:00"
    },
    {
      "name": "Storage",
      "healthy": true
    }
  ]
}

/health/dependencies Returns a JSON object describing whether the servers the synchronizer depends on are healthy or not.

{
  "serviceStatus": "healthy",
  "dependencies": [
    {
      "service": "https://telemetry.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528262-03:00"
    },
    {
      "service": "https://auth.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528264-03:00"
    },
    {
      "service": "https://sdk.split.io/api/version",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528265-03:00"
    },
    {
      "service": "https://events.split.io/api/version",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528266-03:00"
    },
    {
      "service": "https://streaming.split.io/health",
      "healthy": true,
      "healthySince": "2021-11-20T19:04:46.528266-03:00"
    }
  ]
}

Admin Dashboard​

Split-sync has a web admin UI out of the box that exposes all available endpoints. Browse to /admin/dashboard to see it.

The dashboard is organized in four sections for ease of visualization:

  • Dashboard: Tile-sorted summary information, including these metrics:
    • Uptime: Uptime metric
    • Healthy Since: Time passed without errors
    • Logged Errors: Total count of error messages
    • SDKs Total Hits: Total SDKs requests
    • Backend Total Hits: Total backend requests between split-sync and Harness servers
    • Cached Feature flags: Number of feature flags cached in memory
    • Cached Segments: Number of segments cached in memory
    • Impressions Queue Size: shows the total amount of Impressions stored in Redis (only Producer Mode).
    • Impressions Lambda: shows the eviction rate for Impressions (only Producer Mode).
    • Events Queue Size: shows the total amount of Events stored in Redis (only Producer Mode).
    • Events Lambda: shows the eviction rate for Events (only Producer Mode).
    • SDK Server: displays the status of Split server for SDK.
    • Events Server: displays the status of Split server for Events.
    • Streaming Server*: displays the status of Split streaming service
    • Auth Server*: displays the status of Split server for initial streaming authentication
    • Telemetry Server*: displays the status of Split server for telemetry capturing.
    • Storage: (only Sync mode) displays the status of the storage.
    • Sync: displays the status of the Synchronizer.
    • Last Errors Log: List of the last 10 error messages
  • SDK stats: Metrics numbers and a latency graph, measured between SDKs requests integration and proxy.
  • Data inspector: Cached data showing feature flags and segments; filters to find keys and feature flag definitions.
  • Queue Manager: expose sizes of Impressions and Events queues.

Dashboard refresh rate

The dashboard numbers are committed every 60 seconds. The Logged Errors, Last Errors Log tiles, and the Data inspector section are populated each time the dashboard is refreshed. For Impressions and Events Queue size the numbers are refreshed every 10 seconds.

Service shutdown​

The split-sync service can catch a kill sig command and start a graceful shutdown, flushing all cached data progressively. Additionally, you can perform graceful stop and force stop (kill -9) with one click from the admin dashboard.

If you have configured a Slack channel and the Slack Webhook URL, an alert is sent to the channel when and initialization or shutdown is performed.

Troubleshooting​

Synchronizer returns 500 HTTP error when used in proxy mode​

When using Synchronizer in proxy mode, initializing an FME SDK factory that connects to the Synchronizer instance never completes. The SDK never becomes ready and receives a 500 HTTP error.

Synchronizer’s debug logs show successful calls to the Harness FME servers, but the JSON response contains an empty list of feature flags:

SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:51 logger.go:35: GET |500| 285.71µs | 10.10.6.249 | /api/splitChanges
SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:52 client.go:60: Authorization [ApiKey]:  1c9s...e19o
SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:52 client.go:56: [GET]  https://sdk.split.io/api/splitChanges?since=-1
SPLITIO-AGENT  - DEBUG - 2020/10/12 21:41:52 client.go:64: Headers: map[Accept-Encoding:[gzip] Content-Type:[application/json]]
SPLITIO-AGENT  - VERBOSE - 2020/10/12 21:41:52 client.go:95: [RESPONSE_BODY] {"splits":[],"since":-1,"till":-1} [END_RESPONSE_BODY]

This indicates that no feature flags are present in the environment associated with the SDK API key used by the Synchronizer. Although the upstream HTTP call succeeds, Synchronizer returns a 500 error to the SDK because it cannot compute any treatments without any feature flags

  • Ensure that the environment linked to the SDK API key used by Synchronizer has active feature flags configured.
  • Alternatively, verify that you are using the correct SDK API key for the intended environment with feature flags.

SDK getTreatment Always Returning 'control' When Using Synchronizer Docker​

After installing and running the Split Synchronizer Docker instance with a Redis instance, and configuring an SDK to use Redis, the getTreatment call always returns 'control'.

By default, the Synchronizer Docker instance uses a prefix for Redis keys. If the SDK does not specify the same prefix in its Redis configuration, it cannot read the data stored by Synchronizer.

  1. Check if the Synchronizer is using a prefix by running:

    redis-cli
    keys *

    Look for text before "SPLITIO" in the keys.

    Example output showing the prefix myprefix:

    127.0.0.1:6379> keys *
    1) "myprefix.SPLITIO.split.Split1"
    2) "myprefix.SPLITIO.splits.till"
    3) "myprefix.SPLITIO.split.Split2"
    4) "myprefix.SPLITIO.split.nico_test"
    5) "myprefix.SPLITIO.split.coach_matching_v1"
    6) "myprefix.SPLITIO.split.clients_on"
    7) "myprefix.SPLITIO.split.Split3"
    8) "myprefix.SPLITIO.split.sample_feature"
    9) "myprefix.SPLITIO.segments.registered"
    10) "myprefix.SPLITIO.split.Demo_split"
    11) "myprefix.SPLITIO.split.clients"

  2. Update your SDK configuration. Specify the redisPrefix parameter in your SDK configuration so it matches the Synchronizer prefix.

    For example, in the Python SDK:

    from splitio import get_factory

    config = {
        'redisHost': 'localhost',
        'redisPort': 6379,
        'redisDb': 0,
        'redisPassword': 'somePassword',
        'redisPrefix': 'myprefix'
    }

No Impressions Sent from Python SDK 7.x and Synchronizer 1.x​

When using Synchronizer 1.x with Python SDK 7.x, the Python SDK processes treatments correctly and Synchronizer does not report any errors. However, no impressions are sent to the Harness FME servers.

Starting in Python SDK 7.0.0, design changes were made to align with the enhancements introduced in Synchronizer 2.0. Therefore, when using Python SDK 7.x, you must upgrade to Synchronizer 2.x.

Why do I see a "POST method: Status Code: 404 - 404 Not Found" Synchronizer error?​

After starting the Split Synchronizer process (version 1.6.0 and above), the Synchronizer debug log and the Synchronizer admin dashboard show the error below on all its network POST calls: POST method: Status Code: 404 - 404 Not Found.

This error occurs because an incorrect API key is passed to the Synchronizer, causing the Synchronizer to be unable to find the Account in the Harness FME servers.

  1. Verify the API key used by Synchronizer is correct. Synchronizer API key must be of SDK type. API keys can be viewed from Admin settings on the API keys page: https://app.split.io/org/[Your Account ID]/admin/apis.

  2. Ensure the API key is properly passed to Synchronizer.

    Common ways include:

    • Command line argument: -api-key <APIKEY>

    • JSON configuration file: The apiKey property is used to issue requests against Harness FME servers.

    • Proxy mode usage: When Synchronizer is used in proxy mode (not Redis), the "auth" section and "sdkAPIKeys" allow setting custom API keys for internal use, enabling SDKs to use the internal custom API key.

    • Docker environment variable: If running Synchronizer within the packaged Docker image, use:

      -e SPLIT_SYNC_API_KEY=<APIKEY>