How to configure Swift protocol with ECS

Check this out  https://community.emc.com/docs/DOC-27683

Hi,

Can you please take a look at this community post?Getting Started with ECS SDKs

Depending on the language and SDK you're using, the configurtion/setup will vary.  This should at least get you started with making some basic calls into the ECS object store.  Please post back if you have further questions or issues.

Ben

Thanks, Can we use ECS without depending on any SDK or third party tool like Cyberduck?

For example the ports open for Swift Protocol is 9024 and 9025 right. Just using a browser and AD authentication on a windows client is it possible??? SDK mostly is for developers is it not???

Thanks, but this one is using Cyberduck which is a free ware third party tool. We need to achieve this in enterprise level

I was looking for, like  If I have a Windows client using Swift Protocol how can I access the NameSpace-->Buckets on ECS Object storage. If we need to depend Cyberduck or Openstack for every thing with a token validation. it is disastrous for an enterprise environment.

No, using Cyberduck should only be for testing.  You can use any HTTP client that's capable of submitting/receiving requests/responses.  The SDKs are useful because they've already been written and tested to perform all the REST interactions available through the various protocols (Swift, S3, Atmos, CAS).  Whatever client you use, you need to conform to the API spec.  I don't know of any way to use a straight browser to work with the APIs.  When using S3, every request needs to be signed using a secret key and this would be difficult to do in a browser.  A couple suggestions might be to try straight curl for Swift and s3curl for S3.  Those two CLI tools work very well for simple object storage interactions.

Amazon S3 Authentication Tool for Curl : Sample Code & Libraries : Amazon Web Services

For Swift, here's a couple examples on how to use straight curl commands (swauth).

$ export SWIFT_ENDPOINT=http://10.4.0.102:9024

$ curl -I -s -H "X-Auth-User: bob" -H "X-Auth-Key:Password1" $SWIFT_ENDPOINT/auth/v1.0

-SET SWIFT TOKEN FROM RESPONSE HEADERS ABOVE

$ export SWIFT_TOKEN=58e86072ad2d41f792f80cfc5af23bd8

-GET CONTAINER

$ curl -I -s -H "X-Auth-Token: $SWIFT_TOKEN" -H "Accept:application/json" $SWIFT_ENDPOINT/v1/namespace1/group1

Ben,

Thanks for your reply, more clear now, but our case is this.

We would like a service account setup that has access to a Swift Object Storage Name space Optimally this namespace would have approximately 25GB of storage allocated so we can experiment with uploading some larger scientific files that are produced by the Basic Science research cores.

The service account needs full authorization to create/delete Swift containers (i.e. buckets) in this namespace. Once configured, we need account details and the APIs base endpoint URL We will be accessing these sandbox APIs using the Swift python CLI, as well as other scripting/language stacks, such as bash, Ruby, etc...

Nope we are using AD authentication with Swift Protocol -- No Openstack used here so no question of Keystone right.

Please clarify

David, /  Ben,

Can you share some Curl statements if you are using AD authentication instead of keystone with the same above setup

David,

Thanks, We are using AD credentials to login to the Linux client with a RDP gateway server. No keystone Auth is used. Almost the Swiftstack setup you mentioned in your email.

Can you  help with some Curl statements please. I mean to connect and then to create a folder, and also put a file on that folder and delete the file and folder. (on Swift-ECS-Linux client)

Under the assumption that you've configured Keystone Auth provider in ECS, here are the general steps using curl showing how to authenticate, create a container, create an object, delete the object and finally delete the container.

If you've configured AD in your Keystone server, I believe you can use AD credentials here.  However, I'm using credentials for a user stored directly in openstack.

You will first need to submit an V3 auth request into Keystone so you can get a token and Swift endpoints (notice the swift details returned in the response which have been preconfigured as object-store endpoints in openstack):

ben@USENSCHUMBLk -X POST http://10.1.83.114:5000/v3/auth/tokens -H "Accept:application/json" -H "Content-Type:application/json" --data-binary "{\"auth\": {\"identity\": {\"methods\": [\"password\"],\"password\": {\"user\": {\"id\": \"92164d50feb74a8293e4ba898bc9720b\",\"password\": \"secret\"}}}, \"scope\": {\"project\": {\"id\": \"a49ec368f7d8495aba7f32e969b000a5\"}}}}" -v | json_pp

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.114...

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.1.83.114 (10.1.83.114) port 5000 (#0)

> POST /v3/auth/tokens HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.114:5000

> Accept:application/json

> Content-Type:application/json

> Content-Length: 201

>

} [data not shown]

* upload completely sent off: 201 out of 201 bytes

< HTTP/1.1 201 Created

< Date: Tue, 12 Sep 2017 20:48:28 GMT

* Server Apache/2.4.7 (Ubuntu) is not blacklisted

< Server: Apache/2.4.7 (Ubuntu)

< X-Subject-Token: 38e30a3ff2f94b86ba3bd131530be610 <--- THIS IS YOUR TOKEN TO USE IN SUBSEQUENT REQUESTS

< Vary: X-Auth-Token

< x-openstack-request-id: req-0185d4c5-a430-4ddf-9076-c9dfe515f0d2

< Content-Length: 2236

< Content-Type: application/json

<

{ [data not shown]

100  2437  100  2236  100   201  16996   1527 --:--:-- --:--:-- --:--:-- 17888

* Connection #0 to host 10.1.83.114 left intact

{

   "token" : {

      "user" : {

         "id" : "92164d50feb74a8293e4ba898bc9720b",

         "domain" : {

            "name" : "Default",

            "id" : "default"

         },

         "name" : "ecs"

      },

      "audit_ids" : [

         "mXgab1f_RjqXtJfNM_aJDQ"

      ],

      "project" : {

         "id" : "a49ec368f7d8495aba7f32e969b000a5",

         "domain" : {

            "name" : "Default",

            "id" : "default"

         },

         "name" : "ecs"

      },

      "issued_at" : "2017-09-12T20:48:28.095526Z",

      "expires_at" : "2017-09-12T21:48:28.095501Z",

      "roles" : [

         {

            "name" : "admin",

            "id" : "20db6bed29c94176af912ed58ac690e4"

         }

      ],

      "methods" : [

         "password"

      ],

      "catalog" : [

         {

            "name" : "keystone",

            "endpoints" : [

               {

                  "interface" : "internal",

                  "region" : "RegionOne",

                  "url" : "http://10.1.83.114:5000",

                  "id" : "5e54af70abfc4736b96af86b36ade958",

                  "region_id" : "RegionOne"

               },

               {

                  "region_id" : "RegionOne",

                  "id" : "6938a8c0ca444b86b8d7407a7bff9fcf",

                  "url" : "http://10.1.83.114:35357",

                  "interface" : "admin",

                  "region" : "RegionOne"

               },

               {

                  "region_id" : "RegionOne",

                  "id" : "8b58ad5f9b0044e69f452a36f7bf5e58",

                  "url" : "http://10.1.83.114:5000",

                  "region" : "RegionOne",

                  "interface" : "public"

               }

            ],

            "type" : "identity",

            "id" : "628d2673f3d040b5b9133852b0cb1714"

         },

         {

            "name" : "glance",

            "endpoints" : [

               {

                  "url" : "http://10.1.83.118:9292",

                  "region" : "RegionOne",

                  "interface" : "admin",

                  "region_id" : "RegionOne",

                  "id" : "0dfaba34164d45539ccca2bda5947d3a"

               },

               {

                  "region_id" : "RegionOne",

                  "id" : "6a37a22612fc45409d24d4135a618bac",

                  "url" : "http://10.1.83.118:9292",

                  "region" : "RegionOne",

                  "interface" : "internal"

               },

               {

                  "region" : "RegionOne",

                  "interface" : "public",

                  "url" : "http://10.1.83.118:9292",

                  "id" : "d546279c9ded4ad7ab2c5319af074a5e",

                  "region_id" : "RegionOne"

               }

            ],

            "id" : "7f1d027f193241ef9ad8b597f5993565",

            "type" : "image"

         },

         {

            "id" : "841722108ac242be888aaace6fb59ad6",

            "type" : "object-store",

            "name" : "swift",

            "endpoints" : [

               {

                  "interface" : "admin",

                  "region" : "RegionOne",

                  "url" : "http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5",

                  "id" : "12ee1fecc0084bdd82b3a708d305252e",

                  "region_id" : "RegionOne"

               },

               {

                  "url" : "http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5",

                  "interface" : "internal",

                  "region" : "RegionOne",

                  "region_id" : "RegionOne",

                  "id" : "46e348d76d7c48b5bd84297b1fb255af"

               },

               {

                  "interface" : "public",

                  "region" : "RegionOne",

                  "url" : "http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5",  <--- THIS IS YOUR ENDPOINT TO USE IN SUBSEQUENT REQUESTS TO SWIFT

                  "id" : "90c4dd5b172542b6bf5f11b8ad1e61dd",

                  "region_id" : "RegionOne"

               }

            ]

         }

      ]

   }

}

Using your token from above, try to list the container in your project/namespace:

ben@USENSCHUMBL4C:~$ curl -H "X-Auth-Token: 20db6bed29c94176af912ed58ac690e4" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5 -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> GET /v1/a49ec368f7d8495aba7f32e969b000a5 HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> X-Auth-Token: 20db6bed29c94176af912ed58ac690e4

>

< HTTP/1.1 401 Unauthorized

< Date: Tue, 12 Sep 2017 20:53:34 GMT

< Content-Type: text/plain;charset=utf-8

< x-emc-request-id: 0a015333:15e70b78334:19c5:0

< Content-Length: 12

<

* Connection #0 to host 10.1.83.51 left intact

Unauthorizedben@USENSCHUMBL4C:~$ curl -H "X-Auth-Token: mXgab1f_RjqXtJfNM_aJDQ" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5 -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> GET /v1/a49ec368f7d8495aba7f32e969b000a5 HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> X-Auth-Token: mXgab1f_RjqXtJfNM_aJDQ

>

< HTTP/1.1 401 Unauthorized

< Date: Tue, 12 Sep 2017 20:54:19 GMT

< Content-Type: text/plain;charset=utf-8

< x-emc-request-id: 0a015333:15e70b78334:19c5:2

< Content-Length: 12

<

* Connection #0 to host 10.1.83.51 left intact

Unauthorizedben@USENSCHUMBL4C:~$ curl -H "X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5 -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> GET /v1/a49ec368f7d8495aba7f32e969b000a5 HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610

>

< HTTP/1.1 200 OK

< Date: Tue, 12 Sep 2017 20:58:56 GMT

< X-Timestamp: 1464119495

< Accept-Ranges: bytes

< X-Account-Bytes-Used: 0

< X-Account-Container-Count: 2

< X-Account-Object-Count: 0

< x-emc-request-id: 0a015333:15e70b78334:19cc:0

< X-Trans-Id: tx0a01533315e70b7833419-cc000000000

< Content-Type: text/plain

< Content-Length: 18

<

ben-keystone

ben-test

* Connection #0 to host 10.1.83.51 left intact

Create a new container:

ben@USENSCHUMBL4C:~$ curl -X PUT -H "X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> PUT /v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610

>

< HTTP/1.1 201 Created

< Date: Tue, 12 Sep 2017 21:08:07 GMT

< x-emc-request-id: 0a015333:15e70b78334:19d7:0

< X-Trans-Id: tx0a01533315e70b7833419-d7000000000

< Content-Type: text/html

< Location: http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl

< Content-Length: 0

<

* Connection #0 to host 10.1.83.51 left intact

Create a new object in the container:

ben@USENSCHUMBL4C:~$ curl -X PUT -d "Hello World ..." -H "Content-Type: text/html; charset=UTF-8" -H "X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl/helloworld.txt -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> PUT /v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl/helloworld.txt HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> Content-Type: text/html; charset=UTF-8

> X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610

> Content-Length: 15

>

* upload completely sent off: 15 out of 15 bytes

< HTTP/1.1 201 Created

< Date: Tue, 12 Sep 2017 21:10:19 GMT

< x-emc-request-id: 0a015333:15e70b78334:19d9:0

< X-Trans-Id: tx0a01533315e70b7833419-d9000000000

< ETag: 8da475e8cbd2eb156ab3766b2eebd69e

< Content-Type: text/html;charset=utf-8

< Last-Modified: Tue, 12 Sep 2017 21:10:19 GMT

< X-Timestamp: 1505250619

< x-emc-mtime: 1505250619628

< Location: http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl/helloworld.txt

< Content-Length: 0

<

* Connection #0 to host 10.1.83.51 left intact

Read the object back:

ben@USENSCHUMBL4C:~$ curl -H "X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl/helloworld.txt -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> GET /v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl/helloworld.txt HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610

>

< HTTP/1.1 200 OK

< Date: Tue, 12 Sep 2017 21:11:02 GMT

< x-emc-request-id: 0a015333:15e70b78334:19d9:c

< X-Trans-Id: tx0a01533315e70b7833419-d9c00000000

< Accept-Ranges: bytes

< X-Timestamp: 1505250619

< ETag: 8da475e8cbd2eb156ab3766b2eebd69e

< Last-Modified: Tue, 12 Sep 2017 21:10:19 GMT

< x-emc-mtime: 1505250619628

< Content-Type: text/html;charset=utf-8

< Content-Length: 15

<

* Connection #0 to host 10.1.83.51 left intact

Hello World ...

Delete the object:

ben@USENSCHUMBL4C:~$ curl -X DELETE -H "X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl/helloworld.txt -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> DELETE /v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl/helloworld.txt HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610

>

< HTTP/1.1 204 No Content

< Date: Tue, 12 Sep 2017 21:11:55 GMT

< x-emc-request-id: 0a015333:15e70b78334:1a3b:0

< X-Trans-Id: tx0a01533315e70b783341a-3b000000000

< Content-Type: text/plain

< Content-Length: 0

<

* Connection #0 to host 10.1.83.51 left intact

Delete the container:

ben@USENSCHUMBL4C:~$ curl -X DELETE -H "X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610" http://10.1.83.51:9024/v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl -v

* Hostname was NOT found in DNS cache

*   Trying 10.1.83.51...

* Connected to 10.1.83.51 (10.1.83.51) port 9024 (#0)

> DELETE /v1/a49ec368f7d8495aba7f32e969b000a5/ben-curl HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 10.1.83.51:9024

> Accept: */*

> X-Auth-Token: 38e30a3ff2f94b86ba3bd131530be610

>

< HTTP/1.1 204 No Content

< Date: Tue, 12 Sep 2017 21:12:35 GMT

< x-emc-request-id: 0a015333:15e70b78334:1a3b:4

< X-Trans-Id: tx0a01533315e70b783341a-3b400000000

< Content-Type: text/html

< Content-Length: 0

<

* Connection #0 to host 10.1.83.51 left intact

Ben,

This works great, was able to put something on the Object storage and doing testing now.

Another question I have is. Domain user AD/LDAP on ECS, how can we access the Object using the curl statements,

For example how do I get the X-Auth Token? using my AD/LDAP credentials.

In my latest example above, you'll see that the IP of the Keystone server (10.1.83.114:5000) is different than the Swift (ECS) endpoint (10.1.84.51:9024).  To be able to accomplish what you're asking, it's necessary to configure your Keystone server to use AD as an authentication provider.  Once that configuration is complete, you will be able to obtain an X-Auth-Token from Keyston using AD credentials.  This configuration is outside the scope of ECS.  Below I've included a couple links to help you get started configuring Keystone with AD (if you haven't already done so).

OpenStack Docs: Integrate Identity with LDAP

HowtoIntegrateKeystonewithAD - OpenStack

Integrate Active Directory with OpenStack Keystone - YouTube

Thanks,

Ben

Ben

We are not using the keystone auth at all. we are creating local user on EMC and accessing it using curl statements. We created one user with a namespace and OBJ storage folder. it was working fine until yesterday. Today we had this error.

Any suggestions? It was working until yesterday

linux ~]$ curl -i -k -X GET -H 'X-Auth-Token: ECS_5xxxxxxxxxxxxxxx' https://swift-ecs.xoom.com:9025/v1/users-test/name-swfit

HTTP/1.1 200 Connection established HTTP/1.1 404 Not Found Date: Thu, 21 Sep 2017 20:06:30 GMT x-emc-request-id: 0a12a90b:15dcee2d691:1fe0d:0 Server: ViPR/1.0 Transfer-Encoding: chunked Content-Type: text/html Via: 1.1 vlp-isowsa1.xoom.com:80 (Cisco-WSA/10.1.0-204)

Connection: keep-alive keypool not found

In that case, ECS does have an option for configuring AD as an external auth provider, but it will not work for accessing the data path (S3, Swift, Atmos, etc).  It's more for management API access via System/Namespace administrators.  As a result, it sounds like AD is not a good fit for what you're trying to do with Swift.

In regards to your 404 error, are you sure the container and object still exist in ECS?  It might be useful for you to install and configure a GUI like Cyberduck.  This will allow you to query your containers and objects and provide a second form of access into the object store for testing.

You can download Cyberduck here: https://cyberduck.io/

You can find some information on configuring Cyberduck here: ECS Test Drive – Getting started with Swift and Cyberduck

Thanks,

Ben