Welcome everyone, this ATE session has began. Our experts are now ready to answer any question you post on this thread. Enjoy!

Hello Everyone,

Thanks for the opportunity. I have more than question, hope this is not a problem.

• Big Data Questions:
  • Assuming we are using Hadoop for Big Data Analytics, what are the advantages of using OneFS against Hadoop HDFS?
  • Are there any, in the near future, alternatives to Hadoop? Is there any coming advances to Map-Reduce?
• Object Storage Questions:
  • Up till now, I am not able to understand how Object Storage works. For example, I understand that NTFS stores larger sizes files than FAT, because of the large iNode size. For Object Storage, there are no iNodes. So, how is file written to disk? I mean what exactly is the change that permitted capability to write large files?
  • Besides, if Object Storage calculates the ID from the content, and stores authorization data within the stored object. That means everyone with access to Object Storage can find the object but only authorized users can access it. Is that true?
  • In case what I am thinking is true, isn't this a security leakage?

Sorry for the multiple questions. I understand I might be needing to learn more about the technologies. However, I find it an excellent chance to get replies from the experts. Replies may include references to external resources. Thanks again.

Hi Haitham,

I will let the Experts reply to your questions, but let me recommend the latest October blog posts on the Isilon Community to you about OneFS and Hadoop, and Splunk as an alternative to Hadoop.

Isilon 

Happy reading.

Niki

Hello Ryan Peterson 

Honestly, those are the answers I was looking for. You really have no idea how much you helped me with your answer. Thank you for the thorough answer, and the effort and time you took to provide this answer. Actually, I have more questions, hope you don't mind.

• How OneFS provides protection, if not through multiple-write (3x in case of HDFS)?
• Is there any technical documentation about OneFS techniques when compared to another file systems? I know OneFS is a proprietary File System, but I hope you have a blog or something that refers to such topics.
• I hope you can share any of your thoughful ideas about "Object Storage". They are really helpful. If you can shed the light on differences between OneFS and Swift (Open Stack) that would be awesome. Even if not, your answer really helped me.

Sorry for the questions. I am in academia, and I am starting my research career in "File Systems". A topic I find very useful actually. Sorry for any inconvenience and thanks for your generosity.

Hi Haithem. Thanks for your question. Let me see if I can address the object portion.

Regardless of whether a file is stored as an object, file or a block, it is technically stored as a contiguous block of data on a disk. File/NAS storage and Object storage are simply abstractions above that process. As far as handling large files, that is precisely what Object storage is designed for. Rather than using a file system with a hierarchical structure, Object stores a file  with both the metadata and raw data packaged together as a unique object. This object is then stamped with a unique identifier and placed in a non-hierarchical bucket. It seems as though you are referencing Content Addressed Storage (CAS). Centera is an example of CAS. With Centera, the application requests to create a new file and the app server sends the file to Centera. Centera performs  the Content Address calculation using a proprietary hash and  sends the address back to application. The application database stores the content address for future reference. The content address is a unique, digital fingerprint that guarantees content authenticity and immutability. When an application needs to access the file, the application only needs to know the content address. The authorization data is not stored within the object. That is governed by the application and the user's privileges at the application layer.

Other object platforms work similarly but use different methods of creating the unique object ID which is stored in an index.

As far as security, each operation is individually authenticated. So, if a user is not authenticated, they will not have permission to access a file. Again, this is done at the application and access control layer.

Access to object storage is via an API, most often a restful API such as Amazon S3,. EMC Atmos or OpenStack Swift.

For a more detailed explanation, here are a few resources for you:

ViPR Services Storage Engine Architecture White Paper

EMC Atmos Cloud Storage Architecture

I understand that one of the questions consistently asked at the Hadoop World booth was "Which Hadoop distribution should I choose?"  

How do our Big Data experts respond to that?

Ryan Peterson  George 

Thank you so much for the shared answers. You are truly the experts. I appreciate your time.

It is really difficult to find great answers online without directly talking to the experts.

Can't wait for the next "Ask the Expert" to take questions to the next level. 

Just after I finish checking the useful resources you have provided.

Thanks

Haitham

Which Hadoop distribution should you use? In the case of ViPR HDFS, EMC gives you the option to choose the Hadoop distribution that best fits your needs. ViPR Services is an object-based unstructured storage engine. ViPR Services supports access to the underlying data via Object APIs such as S3, OpenStack Swift and EMC Atmos. It also provides an HDFS interface to an object bucket. ViPR presents an HDFS-compatible file system. ViPR HDFS provides a client library (ViPR-HDFS Client) that is installed on all the data nodes that run MR jobs on the customer’s Hadoop cluster. As such, the customer can use the distribution of their choice.

When a task running on the datanode needs to read a file, the request will go to the ViPR-HDFS client (the customer will point to viprfs:// as their data source) and the ViPR client will communicate with the HDFS head on the ViPR data node. The ViPR client passes in a authN token that identifies the user to the HDFS Head.

The HDFS head in the ViPR Data node receives requests from the ViPR-HDFS client . The HDFS Head then verifies the user’s identity by authenticating against the KDC. Then it talks to the ViPR Services engine and the controller process running on the node to fetch the requested data once authN and authZ succeed.

Bottom line, the goal of ViPR HDFS is to extend analytic capabilities to additional data sources, for example, a large, PB-scale archive for metadata querying, etc. But you can use your existing Hadoop distribution.

Thank you Haitham. We appreciate your participation. The community is only as valuable as the participants!

This ATE events has ended. We would like to thank all those who participated in this discussion, but special thanks to our experts who took their time from their busy schedule to answers our user's questions.

Cheers.