x

Start investing with a
free Kristal.AI account.

icon

Simple. Secure.

*Error message

5 Mins Read

Basics of Redis and Redis cluster

Wikipedia definition of Redis says Redis is an in-memory data structure project implementing a distributed, in-memory key-value database with optional durability. Redis supports different kinds of abstract data structures, such as strings, lists, maps, sets, sorted sets, HyperLogLogs, bitmaps, streams, and spatial indexes.

What does In-memory data structure mean?
It means all data resides in the server’s main memory, unlike databases such as PostgreSQL, MongoDB, and others that store most data on disk or on SSDs. Most of the operations on these traditional disk-based databases requires a round trip to the disk but in the case of in-memory data stores such as Redis, Memcache all the data is stored in memory itself, so by eliminating the need to access disks, it avoids time delays and can access data in microseconds, result in extremely fast performance.

Let’s discuss the two ways we are using Redis at KRISTAL.AI:
First
, To cache the content which doesn’t change often, We have a lot of pages where data or metrics changes only once a day, so, instead of getting that from the database each time a user wants to access that pages, we get the data from Db for the first time and stores it in Redis, So, for the subsequent hits we can directly serve the data from Redis. As Redis is in Memory Database, this would return data much faster compared to the database. And once a day we get the new data for all the pages, there is a job to clear out entries from Redis, so the user gets updated data all the time.

 

Request Flow Diagram

 

The Second interesting case where it can be used is Rate Limiting. The basic concept is that you want to limit requests to a particular service in a given time period. Let’s say you have a login portal and you want to block a user for 24 hrs if he attempts more than 5 times incorrect password within an hour to avoid a Malicious attack or bot attack. Redis can be used to achieve this, most of the banks and fintech products like KRISTAL.AI are using the same.

How this works:
We have set the max value for all the cases in Redis like for the wrong credentials attempts we have set the value as 5 using

hmset LOGIN:DAILY FAILED 5

 

Now whenever a user enters the wrong credentials, a new entry is getting created in Redis for that user like

redis-master-1:6379> get LOGIN:FAILED:abc@gmail.com:DAILY
-> Redirected to slot [4980] located at redis-slave-1:6380
“1”

 

Now, the value of this counter is 1, which is less than that the max value setup for this pattern, so we allow the user to attempt again.

As, soon as this counter becomes more than 5, the user is blocked for the window mentioned while setting up the key, which in this case in DAILY, and will get a nice message on the screen.

redis-master-1:6379> get LOGIN:FAILED:abc@gmail.com:DAILY
-> Redirected to slot [4980] located at redis-slave-1:6380
“6”

Now, Redis clear this counter once the window period is over like in the HOURLY window case, TTL (Time to Live) is 1hr, so Redis clear this key after 1 hr, similarly, in the case of the DAILY window, Redis clear the key after 24hrs.

Redis Cluster:
Redis Cluster is a distributed implementation of Redis. It provides a way to run a Redis installation where data is automatically sharded across multiple Redis nodes. Redis Cluster also provides a certain degree of high availability during partitions, which is in practical terms the ability to continue the operations when some nodes fail or are not able to communicate. For the Redis cluster to work, all Hash slots must be covered either by a master node or slave node. Don’t worry about the Hash slot, we will discuss this in the next paragraph.

How Data Sharding works in Redis:
Redis Cluster uses a form of sharding where every key is conceptually part of a hash slot. There are a total of 16384 hash slots in the Redis Cluster.
Every node in a Redis Cluster is responsible for a subset of these hash slots, so, for example, Let’s say we have a cluster of 3 nodes, where:

Node A contains hash slots from 0 to 5460.
Node B contains hash slots from 5461 to 10922.
Node C contains hash slots from 10923 to 16383.

 

Now Let’s say we want to add a new node D in our cluster, we need to move some hash slot from nodes A, B, C to D.
Similarly, If we want to remove a node from ou cluster due to some hardware issue, to upgrade the Redis version, or any reason, we just have to simply move the hash slots served by A to B and C. And when node A becomes empty we can remove it from the cluster completely. Since moving hash slots from a node to another does not require stop operations, adding and removing nodes, or changing the percentage of hash slots, does not require any downtime.

Master-slave model:
Now as mentioned above Redis Cluster provides a certain degree of high availability. What does this mean?
Let’s assume in our cluster with nodes A, B, C, Node A fails due to hardware issue, connectivity issue, or any reason, since we no longer have a way to serve hash slots in the range 0–5460. Our cluster breaks, It won’t accept any new writes and read operations.

To solve this problem, we at Kristal.AI are using the Master-slave model. Redis provides the capability to have N-1 additional slave nodes for the N node cluster. We can add a slave node while creating the cluster as well as at a later time.
We are using 1 slave per master. We have 3 node machines, and on each machine, Two Redis process are running on different ports. So, in total, we have 6 Redis process running, out of which 3 are master and 3 are slaves.

While creating a Redis cluster it performs Hash slot allocation:

>>> Performing hash slots allocation on 6 nodes…
Master[0] -> Slots 0 – 5460
Master[1] -> Slots 5461 – 10922
Master[2] -> Slots 10923 – 16383

 

Also, add replicas to each master,

Adding replica redis-slave-2:6380 to redis-master-1:6379
Adding replica redis-slave-3:6380 to redis-master-2:6379
Adding replica redis-slave-1:6380 to redis-master-3:6379

 

Once, its done, we should see output like this:

So, our cluster now contains 3 master nodes A, B, C, and 3 slave nodes A1, B1, C1. Now Node A1 replicates A, and if A fails, the cluster will promote node A1 as the new master and will continue to operate correctly.
However, note that if nodes A and A1 fail at the same time Redis Cluster is not able to continue to operate as again it no longer has a way to serve hash slots in the range 0–5460.

Redis Cluster consistency:
Redis uses asynchronous replication to replicate data from master to slave node. Because of this, under a certain condition, it is possible that Redis Cluster will lose some writes that were acknowledged by the system to the client.
Let’s consider a scenario –

The client writes to the master A.
Master A replies with acknowledgment to the client.
Master A replicates data to its slaves A1.

 

Now, as we see Master node sends an acknowledgment to the client without making sure if the data is replicated successfully to its slave node A1. Since it would be a performance hit if it waits for an acknowledgment from the slave node before replying to the client.
So if the client writes something to A, A acknowledges the write but crashes before being able to send the data to its slave nodes, and the slaves that did not receive the write is promoted to master, write is lost forever. Redis Cluster has support for synchronous writes, implemented via the WAIT command, this makes losing writes a lot less likely, but this usually results in low performance. Basically, there is a trade-off to make between performance and consistency.

Disclaimer

The materials and data contained herein are for information only and shall in no event be construed as an offer to purchase or sell or the solicitation of an offer to purchase or sell any securities in any jurisdiction. Kristal Advisors does not make any representation, undertaking, warranty or guarantee as to the update, completeness, correctness, reliability or accuracy of the materials and data herein. All opinions, forecasts or estimation expressed herein are subject to change without prior notice. Kristal Advisors and its affiliates accept no liability or responsibility whatsoever for any direct or consequential loss and/or damages arising out of or in relation to any use of opinions, forecasts, materials and data contained herein or otherwise arising in connection therewith.

Post tagged under

Other stories you might like

cross
Change your country to India?

You’re now visiting Kristal.AI Worldwide.
To explore diverse offerings in Indian and global products, select India as your country.

ind

India

hkg

Hong Kong

sgp

Singapore

global

Other
Accredited Investor Declaration

This is offered only to Accredited and Institutional Investors as defined under the Securities and Futures Act, Chapter 289 of Singapore (“Act”), which broadly comprises of regulated financial Institutions, large corporates, high net worth individuals and sophisticated investors.

An Accredited Investor is an individual
  • Whose net personal assets exceed in value SGD 2 million (or it’s equivalent in a foreign currency) with value of his/her primary residence capped at SGD 1 million, or
  • Whose financial assets (net of any related liabilities) exceed in value SGD 1 million (or it’s equivalent in a foreign currency), or
  • Whose income in the preceding 12 months is not less than SGD 300,000 (or it’s equivalent in a foreign currency)

I agree to opt-in as Accredited Investor and will submit required documentation to confirm the same.

< Back Proceed as Private Wealth

Wealth Management Solutions
Accredited Investor Declaration

This is offered only to Accredited and Institutional Investors as defined under the Securities and Futures Act, Chapter 289 of Singapore (“Act”), which broadly comprises of regulated financial Institutions, large corporates, high net worth individuals and sophisticated investors.

An Accredited Investor is an individual
  • Whose net personal assets exceed in value SGD 2 million (or it’s equivalent in a foreign currency) with value of his/her primary residence capped at SGD 1 million, or
  • Whose financial assets (net of any related liabilities) exceed in value SGD 1 million (or it’s equivalent in a foreign currency), or
  • Whose income in the preceding 12 months is not less than SGD 300,000 (or it’s equivalent in a foreign currency)

I agree to opt-in as Accredited Investor and will submit required documentation to confirm the same.

Proceed as Private Wealth

KRISTAL PRIVATE MARKET

Easy access to high-potential startups and private companies to diversify and enhance your portfolio.

For investors who want to support innovative businesses and be part of their success stories.

*The range of private investment opportunities will be made accessible post signup.

OVERVIEW

Explore and Identify

Select from a range of expertly curated Pre-IPO deals from across the globe

Efficient Execution

Enabling easy exposure as Kristal.AI takes care of the paperwork involved with a private investment

Only the Best

A ready-made collection of unicorn companies who are challenging incumbents and rewriting the rules of their industry

Success Story

Diversify your portfolio and be part of the success story of a dynamic and rapidly growing unicorn

Contact our Relationship Manager
ALTERNATIVE INVESTMENTS

Diversify your portfolio with exclusive Private Equity, Pre-IPO deals, VC Funds & other investments like Bitcoin, Gold, Hedge Funds and many more.

Kristal fractionalizes difficult-to-access deals and opportunities.

View Alternative Investments

OVERVIEW

Instant Access

A wide range of alternative asset classes such as Gold, Bitcoins, Hedge Funds and many more; readily made available

Risk Management

Enables investments in diversified assets hence spreading the risk

Easy Tracking

Enables you to get a consolidated view of investments in all assets at one place, hence facilitating easy monitoring and tracking

Exclusive Deals

Access to exclusive Venture Capitalist, Startup as well as pre-IPO investment opportunities

Low Investment Tickets

Facilitates fractionalization of investments in deals

Contact our Relationship Manager
FUNDS PLATFORM

Professionally managed funds with active investment strategies, screened for you by our investment committee to diversify your portfolio.

Enabling clients to access exclusive investment strategies.

View Funds

OVERVIEW

Access

Provides Access to Professional Fund Managers

Low Investment Tickets

Facilitates fractionalization of investments in funds

Diversity

A wide spectrum of investment strategies ranging from Arbitrage to Credit to Hedge Funds and many more

Fees

No additional costs are charged by Kristal.AI

Exceptional Performance

Funds with an impressive track record and potential for high profitability

Leverage

Benefits from insights of the expert selection and advisory team

Easy Management

No time consuming management of portfolios

Contact our Relationship Manager
TAILOR-MADE PRODUCTS

Customize products based on your market outlook and specific needs across equities, bonds, gold and more.

For investors looking to enhance portfolio yield based on market views.

*The range of Tailor Made Products will be made accessible, post login

OVERVIEW

Enhanced Earnings

Structures with potential to enhance portfolio yield based on market movements

Customization

Product features can be designed to articulate investor views

Fees

A flat and transparent fee structure like nowhere else

One-Click Access

A single feature ‘Ideas’ enables you to access recently launched and trending market opportunities

Contact our Relationship Manager
BUILD YOUR OWN KRISTAL

Can’t find what you are looking for? Create your own exclusive investment strategy with risk management & periodic reporting.

For seasoned investors who prefer to design their own portfolio.

OVERVIEW

Empowered Investing

Enables clients to create their own custom portfolio based on their unique needs

Unlimited Choice

Access to all possible exchange traded assets

Flexible Management

Flexibility in delegating the portfolio management role to us or actively taking up the day to day investment decisions, in your custom Kristal

Custom Fee Structure

A simple, transparent and a customized fee structure

Access to Expertise

Custom performance reporting as well as advisory support

Contact our Relationship Manager