Unsolved
This post is more than 5 years old
20 Posts
0
1788
June 9th, 2010 05:00
Performance problems when archiving to Centera
I am working on the project to develop a custom application (developed in C++, using Centera SDK 3.2 Patch 5) that will be used for archiving of new and migration of the existing documents from optical storage to Centera storage appliance.
Each document consists of two files that are saved in one Centera Clip as two separate BLOBs. The application is using the EMBEDDING threshold of 100 K.
Majority of the files (75%) are less than 100K. The file size is rarely larger than 1 MB.
EBR and static retention classes are assigned to each new created Clip.
If the document already exists in Centera, the application opens the existing clip, replaces the existing two blobs and writes the clip back (saving the new C-clip).
Target Centera consists of two clusters with bidirectional replication. Cluster version is Gen4 CE+ (Compliance Edition Plus). CentraStar OS version is 4.02.
The module that writes to Centera is a separate thread. The application can start configurable number of these threads.
At this point we are noticing that storing files to Centera is the bottleneck. We can not get better throughput than about 5 to 10 documents per second (10 to 20 files per second) no matter how many threads are used.
Below are the current performance results.
--------------------------------------------------------
Num of New Clip Write Existing Clip Write
threads (files/sec) (files/sec)
--------------------------------------------------------
1 9 6
2 15 11
5 19 12
10 21 14
20 24 15
--------------------------------------------------------
Document per second speed is 1/2 of Files/second
Here are my questions:
1. Is above mentioned archiving speed considered normal? What is the expected archiving speed for the file size sample I described above (75% files sizes in the range of 2K to 100K, 25% 100K to 1MB)?
2. Why is there a considerable difference in speed when writing to an existing clip (re-writing existing clip is slower)?
3. Can you provide more details or point me to the resources that describe the methods to optimize the archiving performance.
We followed the recommendations from the SDK documentation. I understand that the questions might be a bit too general but any help would be greatly appreciated.
EMCDennis
124 Posts
0
June 9th, 2010 07:00
Some comments that may or may not help
- You cannot update exiting clips without writing new clips. Your "clip update" is actually the writing of a new clip thus would fall under the same performance constraints as a new clip write with maybe the only exception being that you are first reading in the old clip thus doing 2 functions instead of a single function (write).
- Under CE+ with retention, all of your old clips that exist prior to "updating" them will be stranded and can only be deleted should retention policy allow it. This will dramatically increase the used object counts as clips no longer referenced by the application remain on the cluster.
- CPP is slower than CPM - check the protection scheme. CPP requires more lookups than CPM
- Check the storage strategy. Performance over Capacity especially if writing small objects. This effects single instancing of data.
- Clip updates on clips containing embedded blobs will cause the rewriting of the embedded blobs. This is a performance hit but also reduces the object count.
- The number of access nodes effects performance. This really comes down to the number of open connections that exist per access node. This should not exceed ~20 per node.
- Network bandwidth greatly impacts performance
- The way the application is written. You should establish a single 'pool open' and then thread everything after this. It is bad form to open the pool, close the pool, open the pool etc. - Then again, I am not a developer
Any reference material that you may have interest in should be found on powerlink
andrej3
20 Posts
0
June 9th, 2010 08:00
Hi Dennis,
Here are my comments:
- You cannot update exiting clips without writing new clips. Your "clip update" is actually the writing of a new clip thus would fall under the same performance constraints as a new clip write with maybe the only exception being that you are first reading in the old clip thus doing 2 functions instead of a single function (write).
A: I am aware that "clip update" actually creates a new clip. The main (and only) reason for opening and updating the existing clip is to "reuse" the retention information set on original clip.
- Under CE+ with retention, all of your old clips that exist prior to "updating" them will be stranded and can only be deleted should retention policy allow it. This will dramatically increase the used object counts as clips no longer referenced by the application remain on the cluster.
A: This is also taken into consideration. The application keeps track of the "old" clips. The updated clip also contains reference to "old" clip. We also do not expect many updates of the documents. Most transactions are for the new documents.
- CPP is slower than CPM - check the protection scheme. CPP requires more lookups than CPM
A: I believe that it is CPM - need to check with storage group.
- Check the storage strategy. Performance over Capacity especially if writing small objects. This effects single instancing of data.
A: How do you check/set the storage strategy? This is probably in the domain of the storage group that is in charge of Centera hardware.
- The number of access nodes effects performance. This really comes down to the number of open connections that exist per access node. This should not exceed ~20 per node.
A: 2 access nodes
- Network bandwidth greatly impacts performance
A: The Centera clusters are on local network.
- The way the application is written. You should establish a single 'pool open' and then thread everything after this. It is bad form to open the pool, close the pool, open the pool etc. - Then again, I am not a developer
A: The application is using a single pool connection that stays open and is shared among the threads.
Given the above, do you think that the performance is "reasonable"?
Can you give the "optimal" performance under above conditions.
I am basically looking for some performance results that I can compare against.
If the best/optimal performance for the given clip storage strategy and file sizes is 10 times better than what I am getting I would like to know.
Thanks,
andrej
Apex_Ops
1 Rookie
•
90 Posts
1
July 29th, 2010 07:00
Hi Andrej,
I see you posted a question regarding performance problems on a centera. There are many possible factors that can cause a performance degradation when archiving data to a centera. May I suggest that you open up a service request with EMC Centera Support to assist you with this issue? Can you also reply to the forum once you have an update/solution?
Thanks for your assistance and participation in the Support Forums.
Regards,
Ronan
andrej3
20 Posts
0
July 29th, 2010 12:00
I posted the same question in Centera SDK forum where I got the answer from Graham Stuart (https://community.emc.com/message/486507#486507).
Here is Graham answer:
It turns out that this problem was related to the use of large embedded blobs causing the CDF buffer to fill up and swap to disc. We increased the size to be large enough to hold all Andrej's embedded data + metadata and the performance improved massively to within expected levels.
andrej3
20 Posts
0
July 29th, 2010 12:00
Hi Ronan,
I posted the same question in Centera SDK forum where I got the answer from Graham Stuart (https://community.emc.com/message/486507#486507).
Regards,
Andrej