Message Streaming Provider

In most cases, the provider is a simple string alias that matches a configuration section from the MESSAGING_CONFIG_NAME variable.

Assuming that an environment's default provider has been configured, the provider alias may be omitted from the configuration.

If the list of providers must be overridden for some reason, the following feature may be leveraged. Instead of using the name/alias of a provider configured on the API server, you may use a special messaging-provider-agnostic URI to embed connection details within the tenant configuration.

The form of the URI is:


activemq:jms://user:pass@test.reltio.com:61616

or

aws://AccessKey:SecretKey@us-east-1

or

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

or

google://client%40email.com:MIICdQIBADANBgkqhki%2FG9w0BAQEFA...@my-project

A valid URI must include the following sections:

The Parameter table explains the URI fields and relevant description of the parameters.
Table 1. Parameters
URI field   Description Example
scheme   This field defines a valid broker type and protocol. Currently, activemq:jms,AWS, Google and Azure are supported. activemq:jms
username and password  
  • ActiveMQ: The username and password (separated by a colon) configured for queue or topic access within the broker.
  • AWS: The access-key and secret key, separated by a colon.
  • The Google Cloud Platform (GCP): The client email and private key (in a plain form with first or last lines with line breaks removed. It is escaped by URI rules), separated by a colon.
  • Azure: The Shared Access Signature (SAS).
user:pass
host and port  
  • ActiveMQ: The host and port of the message broker (separated by a colon).
  • AWS: The AWS region name for the social networking service (SNS) topic/ Simple Queue Service (SQS).
  • GCP: The GCP default project for the PubSub topic.
  • Azure
test.reltio.com:61616

Use this simple Python2 command shown as following:

echo '789secretkey999' | python2 -c 'import urllib, sys;
        sys.stdout.writelines(urllib.quote_plus(l, safe="/\n") for l in
    sys.stdin)'

The following example displays how the Python3 works:

echo -n '789secretkey999' | python3 -c 'import urllib.parse, sys;
        sys.stdout.writelines(urllib.parse.quote_plus(sys.stdin.readline()))'
When you are using URI style configuration, it is important to encode characters that are not legal within URI fields. The most common case is the occurrence of a slash "/" character in Amazon Secret-Keys. Common encodings are listed below:
 
      !              %21
      "              %22
      #              %23
      $              %24
      %              %25
      &              %26
      '              %27
      (              %28
      )              %29
      *              %2A
      +              %2B
      ,              %2C
      -              %2D
      .              %2E
      /              %2F
      :              %3A
      ;              %3B
      <              %3C
      =              %3D
      >              %3E
      ?              %3F
      @              %40
      [              %5B
      \              %5C
      ]              %5D
      {              %7B
      |              %7C
      }              %7D
Note: If the provider URI is not parsed, due to syntax errors or unencoded special characters, an error will be logged. The messages will not be sent to that provider's destinations.

Type

The type of destination may be provider specific. If this is omitted, the default type is queue. Supported types are:

  • queue- supported by both ActiveMQ and Amazon SQS
  • topic- supported by all: ActiveMQ, Amazon SNS, and Google PubSub

Name

The name of the destination can be any arbitrary string.

Character restrictions may be provider specific.

The Provider Restriction table displays the list of providers along with the character restrictions.

Table 2. Provider Restriction
Provider RestrictionJMS The names can be a combination of alphanumeric characters.

For example, the '.' (dot), '-'(dash) or '_' (underscore) characters.

AWS The names can be a combination of alphanumeric characters and the '-' (dash) or '_' (underscore) characters.
Note: The '.' (dot) is invalid in SNS/SQS names.
GCP Names must start with a letter, and includes only the following characters:
  • letters
  • numbers
  • dashes (-)
  • periods (.)
  • underscores (_)
  • tildes (~)
  • percents (%)
  • plus signs (+)
Cannot start with goog.
Azure Names can include letters, numbers, periods (.), hyphens (-), underscores (_), and slashes (/), up to 260 characters. Must start and end with a letter or number. Queue names are also case-insensitive.
Note: It is recommended to use valid name identifiers instead oif simple names:

The Restriction for Queue Name table displays the naming convention for queue names.

Table 3. Restriction for Queue Name
JMS Just a simple string name, there is no special ID format.
AWS SNS/SQS ARN in the following format: arn:aws:sns:<region>:<account>:<name> / arn:aws:sqs:<region>:<account>:<name>
GCP GCP resource identifier for PubSub topics: projects/{project}/topics/{name}
Azure Azure resource identifier for queues or topics: <namespace>:{name}

The naming convention is observed when assigning destination names to tenants. It is especially important to maintain uniqueness, as multiple tenants with the same destination name will have their data mixed within the queue or topic. A suggested convention is to use a character-separated sequence which includes the destination-type and tenant-id. It is followed by a descriptive queue-name.

The example for a JMSis: queue.TestTenant.allEvents. The example for an SQS is: queue_TestTenant_allEvents.