Message Queue

Message Queue is included and provisioned with your Reltio subscription as an AWS SQS queue, AWS SNS or GCP PubSub topic for each tenant.

As needed, a process setup by the customer will subscribe to the queue/topic. The process receives messages from the queue/topic or queries the stream any time to pull a collection of messages.

Capabilities of the Message Queue/Topic architecture:

  • Events are automatically placed on the queue/topic
  • Each message indicates the event type that triggered the message
  • Only returns IDs, so entity must be fetched
A single queue/topic subscription may have many consumers, but each message will only be delivered to a single consumer.
Note: These semantics are ideal for scaling systems and providing workload distribution where it is desired that the message be processed once and only once.

Best Practices for Message Queue

Connection

To connect to Amazon SQS, use AWS API. For details, refer to SQS Developer Guide.

Note: You must use the appropriate region name for connection. Please contact the Implementation team at Reltio for more information.

To configure SNS topic subscriptions, use your AWS account. For details, refer to Guide to accessing SNS. Passive SNS subscriptions are backed by SQS queues, so pulling from them is the same thing as pulling from SQS queues. For details on how to consume messages, see Consumer Implementation Examples for SQS.

To configure PubSub topic subscriptions or to learn how to read messages from PULL subscriptions, refer to Understanding subscriptions in PubSub.

Note: The terms "Active" and "Passive" refer to "Push" and "Pull" respectively.

Security

To access AWS SQS queue or AWS SNS topics, you need to provide valid Amazon credentials: AWS Access Key, AWS Secret Key.

To access PubSub topics, you need to provide valid Google credentials: GCP Client email, GCP Private Key.

Messaging Destinations

Each Tenant in Reltio API has its own event queues/topics.

Naming Convention

Recommended naming conventions for queues are similar to (Environment)_(Tenant) or Reltio_Events_(TenantID)_(QueueID). Please contact Reltio implementation team for more information.

Note:
  • SNS topic and SQS queue names can consist of alphanumeric characters, the dash "-" and the underscore "_". The dot "." is NOT supported in SNS/SQS.
  • PubSub topic names must start with a letter, and contain only the following characters: letters, numbers, dashes (-), periods (.), underscores (_), tildes (~), percents (%) or plus signs (+). These names cannot start with "goog".

Message Consumer

Tolerate Duplicate and Out-of-order Messages.

Most highly scalable cloud systems (Amazon SQS/SNS and Google PubSub) do not guarantee message order and may occasionally deliver duplicate messages. Make your processing logic tolerant of duplicate and out-of-order messages. These steps will also ensure you can scale your consumer using multi-threading, discussed later.

Message Delivery Considerations

SQS/SNS and PubSub guarantee that messages will be delivered "at least" once. As a result, you might receive a duplicate copy of the message (at-least-once delivery guaranteed). When using SQS/SNS/PubSub, the design and code should be done in a manner that they are prepared to receive the same message more than once, but only process it one time.

Improve Receive Throughput by Processing Concurrently

Keep in mind SQS queue semantics support multiple consumers on the same queue, with each message delivered to one and only one consumer. This means that you may have consumers processing messages in 'n' different threads, but each message will only be processed once. To create concurrent consumers using SQS, use multiple polling threads.

Understanding Topic Subscriptions

A topic subscription can be passive (PULL in terms of GCP) or active (PUSH in terms of GCP). Passive subscriptions just aggregate messages sent to them and do nothing with the messages until explicitly asked. Such subscriptions can be pulled by a consumer. After processing the messages successfully, the consumer should delete them from subscription with another explicit call. On the other hand, Active subscriptions take some action when they get a new message. Depending upon the platform, the actions taken could be similar to the examples listed below.
  • calling some HTTP endpoint with message content as a payload
  • sending text message to a phone number (SMS)
  • sending email with message content

Optimize Network IO and AWS Cost by Batching

SQS is based on Amazon Web Services (AWS), so it uses an HTTP request to retrieve messages and perform all other SQS tasks. Making fewer network round-trips can improve performance.

PubSub is based on Google Cloud Platform (GCP), so requesting messages in batches (instead of one message at a time) is more optimal.

Filter Out Events that are not needed

Before processing a Reltio Event, you can pre-parse the message body and discard event types that are never going to be used in your integrations.