MySQL NDB Cluster 8.0 – increase memory allocated for Data Nodes

MySQL NDB Cluster is one of HA solution available in MySQL. It uses performance powerful engine NDB.
If you would like to compare capabilities between NDB and InnoDB engine you could read through it here. Ok, MVCC is not friend with NDB engine, but in-memory data but performance related features make impression.
I have been asked recently to check if DataMemory parameter (memory allocated in each Data Node) could be adjusted without entire cluster shutdown. In earlier release (< 8.0) it was not possible. Memory Data on Data Nodes is used for exposed data. So if you don’t have enough memory then
I’ve decided to build environment to check if this is possible. One of my friend asked my to check up note:

How Much Memory Do I Need Per Data Node in MySQL Cluster? (Doc ID 1023123.1)

Yes, it seems to be possible now:

Nowadays, because of containers, very often we could build something much more quicker than it used to be. There is very nice article from one of MySQL Product Manager about setting up mysql cluster using Docker. Image is marked as experimental, but it should be enough for test. Open source enough too (I’ve just realized when reading that there is commercial version too -> more here).
During my try I’ve have below issues:

• Cluster logfile

When tried to start management node following exact command from tutorial I had problem with cluster logfile. It appeared to be directory. Solution this time was easy -> create file on host before starting container.

• Static IP assignment for containers

It’s good idea always to assign statically assign ip to containers using –ip option.

• Problem with –log-error flag for mysql-cluster image

I’m not sure what was the problem, but starting container with –log-error caused my containers crash without any sufficient error logging. I’ve decided to forget custom redirect logging to host and start it without it:

docker run -d --net=cluster_8.0 --name=mysql1 --ip=192.168.0.10 -e MYSQL_RANDOM_ROOT_PASSWORD=true mysql/mysql-cluster mysqld

Let’s check cluster using ndb_mgm utility:

	ndb_mgm> show;
	Cluster Configuration
	---------------------
	[ndbd(NDB)]     4 node(s)
	id=33   @192.168.0.3  (mysql-8.0.30 ndb-8.0.30, Nodegroup: 0, *)
	id=34   @192.168.0.4  (mysql-8.0.30 ndb-8.0.30, Nodegroup: 0)
	id=35   @192.168.0.5  (mysql-8.0.30 ndb-8.0.30, Nodegroup: 1)
	id=36   @192.168.0.6  (mysql-8.0.30 ndb-8.0.30, Nodegroup: 1)
	
	[ndb_mgmd(MGM)] 1 node(s)
	id=254  @192.168.0.2  (mysql-8.0.30 ndb-8.0.30)
	
	[mysqld(API)]   4 node(s)
	id=24   @192.168.0.10  (mysql-8.0.30 ndb-8.0.30)
	id=25 (not connected, accepting connect from any host)
	id=26 (not connected, accepting connect from any host)
	id=27 (not connected, accepting connect from any host)

We could check memory usage for each data node.

	ndb_mgm> 33  report memoryusage
	Node 33: Data usage is 0%(6 32K pages of total 32762)
	Node 33: Index usage is 0%(6 32K pages of total 32762)
	
	ndb_mgm> 34 report memoryusage
	Node 34: Data usage is 0%(6 32K pages of total 32762)
	Node 34: Index usage is 0%(6 32K pages of total 32762)
	
	ndb_mgm> 35 report memoryusage
	Node 35: Data usage is 0%(20 32K pages of total 32760)
	Node 35: Index usage is 0%(8 32K pages of total 32748)
	
	ndb_mgm> 36 report memoryusage
	Node 36: Data usage is 0%(20 32K pages of total 32760)
	Node 36: Index usage is 0%(8 32K pages of total 32748)

Ok. It’s 1G memory as /etc/mysql-cluster.cnf.

After modyfing /etc/mysql-cluster.cnf , restart management node (this seems to be required to reread update config) time to restart each data node:

	33 restart;
	34 restart;
	35 restart;
        36 restart;

When everything is done after quick check cluster log we see failover to replicated block worked well. For example node 33:

2022-09-02 08:48:16 [MgmtSrvr] INFO     -- Node 33: Suma: initiate handover for shutdown with nodes 0000000000000000000000000000000400000000 GCI: 32719
2022-09-02 08:48:16 [MgmtSrvr] INFO     -- Node 33: Suma: handover to node 34 gci: 32719 buckets: 00000001 (2)
2022-09-02 08:48:21 [MgmtSrvr] INFO     -- Node 33: Suma: handover complete

2 replicas across replication group:

[ndbd(NDB)] 4 node(s)
id=33 @192.168.0.3 (mysql-8.0.30 ndb-8.0.30, Nodegroup: 0, *)
id=34 @192.168.0.4 (mysql-8.0.30 ndb-8.0.30, Nodegroup: 0)

Modification on management node seems to work and memory in data node successfully updated.

ndb_mgm>  33 report memoryusage
Node 33: Data usage is 0%(6 32K pages of total 36858)
Node 33: Index usage is 0%(6 32K pages of total 36858)

ndb_mgm> 34 report memoryusage
Node 34: Data usage is 0%(6 32K pages of total 36858)
Node 34: Index usage is 0%(6 32K pages of total 36858)

ndb_mgm> 35 report memoryusage
Node 35: Data usage is 0%(20 32K pages of total 36856)
Node 35: Index usage is 0%(8 32K pages of total 36844)

ndb_mgm> 36 report memoryusage
Node 36: Data usage is 0%(20 32K pages of total 36856)
Node 36: Index usage is 0%(8 32K pages of total 36844)

Yes. Everything ok. Updating memory in rolling fashion works nice.