Unsolved
This post is more than 5 years old
7 Posts
0
3052
June 24th, 2015 08:00
How and when is rebalancing triggered when adding new sds devices?
Hi,
My test setup is 3 servers with 2 devices each.
After creating and filling my thin volumes, each device is filled by 20%.
When adding 1 device to each server no rebalancing happends.
I remove the added devices, disable rebalancing, then add the new devices again.
Then I enable rebalancing again, then rebalancing occurs, but only fills the new devices by 5%.
If I do this again, disable rebalancing, add devices, enable again, then the newest devices gets 2-3%.
After adding more and more devies less happends, until nothing gets rebalanced.
Can I manually force rebalancing?
Manual rebalance by removing devices and aborting removal kinda works but it is not how you want to do.
Only if I create new volumes they are spanned over the new devices.
Dd-ScaleIO
12 Posts
0
June 25th, 2015 06:00
Hi.
Rebalance in ScaleIO is triggered automatically when adding capacity to the Storage Pool.
Having said that, if the amount of data to be moved is small, rebalance won't be triggered.
What is the size of the devices and volumes? Is I/O still running?
BTW, You cannot manually trigger the rebalance.
PowerFlex Mark
40 Posts
0
June 25th, 2015 07:00
Hi,
Normally a rebalance is triggered in ScaleIO each time a new SDS is added/removed.
You may add/remove nodes/disks dynamically, the system automatically migrates and rebalances the data.
Hope this helps,
Best regards,
Mark
boxar
7 Posts
0
June 26th, 2015 02:00
The size of the devices are 1.8 TB but limited to 97.7 GB.
The volume is 32 GB.
Tried to redo all but got similar results.
Started with 3 sds with 1 device each. Created thin volume and filled it.
Added 3 devices to all sds at once this time.
Rebalancing started but stopped after filling the new devices about 10% of the first.
PhysicalDrive10 16.7GB 17.1%
PhysicalDrive11 1.4 GB 1.4%
PhysicalDrive12 1.3 GB 1.3%
PhysicalDrive13 1.3 GB 1.3%
Tried with both I/O running and not.
SanjeevMalhotra
138 Posts
0
June 26th, 2015 07:00
Can you provide the current status of all the Disks as to how much Data in GB each device has?
boxar
7 Posts
0
June 26th, 2015 08:00
I added 2 more devices per sds, no rebalance.
sds1
PhysicalDrive10 16.7GB
PhysicalDrive11 1.4 GB
PhysicalDrive12 1.3 GB
PhysicalDrive13 1.3 GB
PhysicalDrive8 0.0 GB
PhysicalDrive9 0.0 GB
sds2
PhysicalDrive10 16.5GB
PhysicalDrive11 1.4 GB
PhysicalDrive12 1.3 GB
PhysicalDrive13 1.3 GB
PhysicalDrive8 0.0 GB
PhysicalDrive9 0.0 GB
sds3
PhysicalDrive10 16.9GB
PhysicalDrive11 1.3 GB
PhysicalDrive12 1.3 GB
PhysicalDrive13 1.3 GB
PhysicalDrive8 0.0 GB
PhysicalDrive9 0.0 GB
Then...
I expanded the thin volume, from 32 to 56 GB, rebalance started and moved around couple of hundred megs.
After that I expanded the thin volume in 100 GB increments to 1000 GB, rebalance started again and in the end all data was spread out.
Great! But, no more data was added to the volume.
Some sort of minimum size, quite large?
SanjeevMalhotra
138 Posts
0
June 26th, 2015 11:00
If we see, the IOs are spread evenly across all the SDSs.
SanjeevMalhotra
138 Posts
0
June 26th, 2015 12:00
As pointed out by Sean, your IOs are evenly distributed among all the SDS. This seems to me as working as designed. In order to verify, you can add more data and you can verify it is evenly distributed among all the SDS.
alexkh
60 Posts
0
June 28th, 2015 06:00
Can you please list ll your steps including the device size (old and new) and the size of the volumes you've created.
We would like to reproduce this in our lab.
boxar
7 Posts
0
June 29th, 2015 00:00
1. Install SIO with wizard on 3 windows 2012 R2 servers.
2. Add 1 device to each SDS, in my case 1.8 TB SATA.
3. Set Device Capacity Limit to 100000 MB on each device.
4. Create thin volume of 32 GB.
5. Install SDC on another Windows 2012 R2 server and add the 2 MDMs.
6. Map volume to SDC.
7. Run IOmeter to create 5 GB file and add another 5 GB of data files.
8. Add 1 device to each SDS of 1.8 TB, set limit to 100000 MB, some rebalancing occurs.
9. Add 2 device to each SDS of same size and limit, some rebalancing occurs again.
10. Add 2 device to each SDS of same size and limit, no rebalancing.
11. Increase size of thin volume to 54 GB, some rebalance occurs again.
12. Increase size of thin volume to 102 GB, some rebalance occurs again, still only 10 GB data written.
13. Increase size of thin volume to 1000 GB, rebalance starts again and does not stop until every device is balanced within 0.1%.
I run IOmeter after adding new devices and rebalancing has stopped, but not during.
Hope I didn´t miss anything.