Isilon Search can connect to the cluster but Scan fails

I am really struggling to troubleshoot this problem because it just doesn't seem to make any sense.

After deploying the OVA and adding my Isilon cluster as a source, the Access Zones are recognised and the SMB share is mounted on the Isilon Search appliance. I can browse the files and even read the contents, but the Index never populates because the Scan jobs shows up as failed.

Looking at the logs under /usr/local/search/log I see the following:

==> search-adminapi-stdout-issearchvm.log <==

Access zones of Source server isilon.example.com: [{:zone_id=>1, :zone_name=>"System", :system=>true, :zone_path=>"/ifs"}, [...] {:zone_id=>15 [...]}]

*******before api source create

get_source_by_field_value with query json: {"from":0,"size":999,"query":{"bool":{"must_not":null,"must":{"term":{"general.name":"ISILON"}}}},"sort":[{"general.name":{"order":"asc","mode":"min"}}]}

get_source_by_field_value with query json: {"from":0,"size":999,"query":{"bool":{"must_not":null,"must":{"term":{"general.address":"isilon.example.com"}}}},"sort":[{"general.name":{"order":"asc","mode":"min"}}]}

******in get_source_by_id, id:0007430a01a2b8d41d52c91d2ff48784fd9a

mount test on share path //isilon.example.com/zinc$ success

******after audit setting

Access zones of Source server isilon.example.com: [{:zone_id=>1, :zone_name=>"System", :system=>true, :zone_path=>"/ifs"}, [...] {:zone_id=>15 [...]}]

******before add source

******in add source

******add source response: {"_index"=>"system", "_type"=>"source", "_id"=>"0007430a01a2b8d41d52c91d2ff48784fd9a", "_version"=>1, "_shards"=>{"total"=>1, "successful"=>1, "failed"=>0}, "created"=>true}

******aftere add source

*******after api source create

create audit response : {:code=>201, :body=>{"_index"=>"system", "_type"=>"audit", "_id"=>"AWb1Kbg5RkV7iPXNHXWK", "_version"=>1, "_shards"=>{"total"=>1, "successful"=>1, "failed"=>0}, "created"=>true}, :token=>"[..]"}

Get content index info for app isilon success

Creating index ISILON in app isilon

Create index ISILON in app isilon success

set_index_user_roles response: {:code=>200, :body=>{"message"=>"ok"}, :token=>"[...]"}

create audit response : {:code=>201, :body=>{"_index"=>"system", "_type"=>"audit", "_id"=>"AWb1Kbm7RkV7iPXNHXWL", "_version"=>1, "_shards"=>{"total"=>1, "successful"=>1, "failed"=>0}, "created"=>true}, :token=>"[...]"}

Create workitem request: {"general":{"category":"job","type":"add_src","subtype":null,"platform":null,"feature":null,"name":"ISILON","description":"activity for source ISILON","source":["0007430a01a2b8d41d52c91d2ff48784fd9a"],"index":["isilon"],"node":{"target":"_all"},"priority":8,"stoptime":"5000-01-01T00:00:00.000Z","status":"pending","heartbeat":"2018-11-08T21:10:44.482Z","createdtime":"2018-11-08T21:10:44.482Z","modifiedtime":"2018-11-08T21:10:44.482Z"},"owner":{"modifiedby":{"id":"31393137323739382d373733382d313033382d393838332d303964373839336435663739","name":"admin","directory":"DefaultOpenLDAP","dn":"cn=admin,ou=People,dc=example,dc=com"},"createdby":{"id":"31393137323739382d373733382d313033382d393838332d303964373839336435663739","name":"admin","directory":"DefaultOpenLDAP","dn":"cn=admin,ou=People,dc=example,dc=com"}},"schedule":{},"config":null}

Create workitem response: {:code=>201, :body=>{"_index"=>"system", "_type"=>"workitem", "_id"=>"AWb1KboWRkV7iPXNHXWM", "_version"=>1, "_shards"=>{"total"=>1, "successful"=>1, "failed"=>0}, "created"=>true}, :token=>"[...]"}

create audit response : {:code=>201, :body=>{"_index"=>"system", "_type"=>"audit", "_id"=>"AWb1KbpERkV7iPXNHXWN", "_version"=>1, "_shards"=>{"total"=>1, "successful"=>1, "failed"=>0}, "created"=>true}, :token=>"[...]"}

Isilon version is: 8.1.0.4.

Protection policy of Source server isilon.example.com: +2d:1n

inode mirror count is: 3

inode total is : 747709955.0

Total inode count is: 747709955.0, total objects count is: 241196760

And in the worker log:

==> search-isilon-worker-issearchvm.log <==

181|P 1|T [Thread-15]|2018-11-09 09:11:12,543|SourceAddHandler::mountShare|INFO|mount: /mnt/isi/ISILON-ISILON-0007430a01a2b8d41d52c91d2ff48784fd9a/ifs => //isilon.example.com/zinc$ |SourceAddHandler.java(196)|Job Id: AWb1KboWRkV7iPXNHXWM; issearchvm

182|P 1|T [Thread-15]|2018-11-09 09:11:12,551|BaseProvider::declaimSource|INFO|[task source-DEclaim] need=false,DEclaimed=true,max=-1,source-id=0007430a01a2b8d41d52c91d2ff48784fd9a,task-id=AWb1KhYLRkV7iPXNHXWO |BaseProvider.java(633)|Job Id: AWb1KboWRkV7iPXNHXWM; issearchvm

[...]

191|P 1|T [Thread-9]|2018-11-09 09:11:12,794|MonitorWorkerHandler::refreshSource|INFO|Get the source info for id: 0007430a01a2b8d41d52c91d2ff48784fd9a |MonitorWorkerHandler.java(133)|Job Id: AWbwtHE2-2X-a4Sc2mGm; issearchvm

192|P 1|T [Thread-9]|2018-11-09 09:11:12,845|MonitorWorkerHandler::refreshSource|INFO|Get index name: isilon |MonitorWorkerHandler.java(139)|Job Id: AWbwtHE2-2X-a4Sc2mGm; issearchvm

193|P 1|T [Thread-9]|2018-11-09 09:11:12,869|MonitorWorkerHandler::refreshSource|INFO|Get isilon server info: address: isilon.example.com, port: 8080, user: zinc |MonitorWorkerHandler.java(145)|Job Id: AWbwtHE2-2X-a4Sc2mGm; issearchvm

[...]

812|P 1|T [Thread-9]|2018-11-09 09:12:35,668|IsilonAuth::createSessionAuthHeaders|INFO|[cookie][https:// :8080]: [isisessid=3d7dfe4a-dc0d-4453-97ea-f9224ff93cd2]=>[isisessid=8f89ea1b-6984-4c53-a585-5254c8a0121f] |IsilonAuth.java(121)|Job Id: AWbwtHE2-2X-a4Sc2mGm; issearchvm

813|P 1|T [Thread-9]|2018-11-09 09:12:35,670|RestClient::doCommon|WARNING|(1/2) [401] [1ms] [N/A]

[GET],[https://

body=[]

    resp-code=[401]

    resp-body=[{"errors":[{"message":"authorization required"}]}]

|RestClient.java(674)|Job Id: AWbwtHE2-2X-a4Sc2mGm; issearchvm

814|P 1|T [Thread-9]|2018-11-09 09:12:35,670|MonitorWorkerHandler::getItemAttribute|WARNING|Cannot logon to isilon server isilon.example.com with username zinc password ********, will refresh source and have another try |MonitorWorkerHandler.java(246)|Job Id: AWbwtHE2-2X-a4Sc2mGm; issearchvm

Which makes no sense because I can definitely login to the Isilon with the user and password I created, as well as fetching the JSON response from the API using curl:

curl 'https:// :8080/namespace/ifs/SkypeShare/2-BackupService-11/BackupStore/TaskAssignment/BackupTask/BackupTask.txt?metadata' -H 'Connection: keep-alive' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'Authorization: Basic [TOKEN]' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36' -H 'DNT: 1' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-NZ,en;q=0.9,en-GB;q=0.8,en-US;q=0.7,fr;q=0.6,es;q=0.5' --compressed --insecure

{

"attrs" :

[

{

"name" : "is_hidden",

"value" : false

},

{

"name" : "size",

"value" : 40

},

{

"name" : "block_size",

"value" : 8192

},

[...]

The permissions for the zinc user in the Isilon source are exactly as the Installation docs describe. What could be missing here?

Also, my sincere apologies for tagging you directly. I don't know exactly where to post this, so I tagged you because you posted about Search before (based on the (few) results I found).

dfelt

JamesTMorton