MULTIPLE TERMINALS FREEZING AT THE SAME TIME (THEN GOING BACK ONLINE)

Hi guys,
I would like to go back to my issue with freezing Wyse clients I reported some time ago here: https://www.dell.com/community/Wyse-ThinOS/V10LE-T10-FREEZING-AND-CRASHING-XENAPP4-5-7-x-NET-APPS/td-p/5173008, as I have more details on what's happening with them.

I am desperate to find a workaround (or fix) as this disrupts production and causes >£10k loses to the business I work for.

In two separate warehouses with 500 Wyse clients each, 2-3 years we started to experience global freeze of terminals where multiple of them go offline at the same time: no trap error on screen, no power to USB ports to obtain a memory dump or connect a keyboard, screen shows the last state of the Citrix published .NET app before the freeze. The only solution is to physically power cycle frozen terminals which then connect to their applications fine and never freeze again during this one instance of the global freeze.

Freezing seems to happen only to terminals hosting particular set of .NET applications, we have not seen this on clients with other apps (i.e. Chrome) or published desktops (Windows workstations). It is not triggered by our warehouse staff as some frozen terminals are not in use at the time of the incidents - there is no interaction between them and the users.

What's important, after the last months of investigation it seems like when multiple terminals freeze, SOME OF THEM GO BACK ONLINE after 30-45min, i.e. they automatically reboot themselves. We also have seen a single terminal which went back online without auto rebooting (with "Connection Reset by the peer" in Event Log), so it is not consistent (but could be a separate issue as well).

Hopefully the above behavior will give you guys a clue what may be causing the terminals to go offline and be unresponsive for 30-45min. What they could be doing during that time? Maybe one of the services is causing this which can be disabled? I suppose DEV team would know if there are any conditions/tasks in ThinOS which could automatically restart terminals in some circumstances.
Please note, while we use Trapreboot=yes feature, the above issue is happening ALSO on old terminals with FW version and Wyse models which don not yet support Trapreboot=yes.

Obviously, besides multiple of our terminals freezing at the same time (every couple of months) we have random, single (1-2) every day freezes with the same symptoms as I described above (i.e. no Trap error). I am not sure if single freezes have the same trigger as the "global" ones but they don't impact production as much.

The above issue is not specific to any model of Wyse client or any ThinOS version. Over the last two years we upgraded firmware on all our clients (from 6.x to 8.6) while migrating from Citrix PS 4.5 to XenApp 7.15 but it is still happening every couple of months. While most of our clients are EOL, the issue affects current firmware for 3010. We don't have enough 3040 clients yet to confirm if this also affects them.

So far Wyse Support in the UK nor in Germany were unable able to help us as they are not aware of similar problems reporting by other customers. I hope that some admins present on this forum, managing large environments, have seen this as can advise. It looks like an issue specific to the environment in my company.

I appreciate your help guys.

-----------
Citrix PS 4.5 - XenApp 7.15 (separate Citrix farm in both warehouses)
Terminals affected: V10L, V10LE, C10LE, T10 (3010) with firmware 6.x - 8.6
ESXi 5 - 7.6
WDM 4.7.2 & 5.5 (separate in both warehouses)
WS 2003 - 2012
.NET Citrix published applications
USB peripherals in use:
- Barcode reader: Dataman DM362 & DMA-CCM-1 I/O
- Symbol and Zebra barcode scanners
- Custom made USB keyboard

------------------------

;*************************************************************
;* General 1 *
;*************************************************************

; Upgrade Settings
; 0 — Disable checking for image
; 1 — (default) Enable a forced firmware upgrade/downgrade process
; 2 — Enable a comparison/non-forced upgrade only process
autoload=2 LoadPkg=0 VerifySignature=no

; If you are using a version of WTOS code < 7.1_133 please use:
; Rapportserver= instead of WDMServer=
;WDMServer=""
;CCMEnable=no
AdminMode=yes Admin-Username=GEN Admin-Password=
Privilege=High KeepDHCPRequestIP=yes TrapReboot=yes

;*************************************************************
;* General 2 *
;*************************************************************

FormURL=ff.jpg
;Reboot=yes Time=17:00-17:10 Wday=Sunday Idle=30
; EnableCacheIni is supported only on T10 and T10D
EnableCacheIni=yes
MirrorFileServer=yes
PlatformConfig=all EOLWarning=no

;*************************************************************
;* General 3 *
;*************************************************************

Autopower=yes
Shutdowncount=0
ShutDown=turnoff
SysMode=Classic
;AddCertificate=O.crt
;AddCertificate=L.crt

;*************************************************************
;* Peripherals *
;*************************************************************

Language=Uk
Device=Keyboard NumLockOff=yes

;*************************************************************
;* Display *
;*************************************************************

; If you are using a version of WTOS code < 7.0.1_13 please use:
; Resolution=1024x768 Refresh=60
Screen=1 Resolution=1024x768
DeskColor="0 0 0"
;DesktopColorDepth=32
Desktop=background.bmp Layout=Center
Screensaver=0
ShutDownInfo=yes

;*************************************************************
;* Time *
;*************************************************************

Timeserver=ntp.com
; If you are using an older version of WTOS code please use:
; TimeZone= 'GMT' ManualOverride=yes Daylight=yes Start=030507 End=100507

TimeZone='Greenwich Mean Time' ManualOverride=yes Daylight=yes Start=030507 End=100507 TimeZoneName="GMT Standard Time" DayLightName="GMT Daylight Time"

;*************************************************************
;* Network *
;*************************************************************

Device=Ethernet Speed="Auto"
; If you are using a version of WTOS code < 7.1_133 please use:
; RapportDisable= instead of WDMService=
;WDMService=Yes SecurityMode=Low
BootpDisable=yes
WakeOnLan=yes
SecurityPolicy=low
DhcpExpire=Reboot
SignOn=Yes ConnectionManager=Maximize LastUsername=yes SaveLastDomainUser=yes LockTerminal=no AutoConnectTimeout=30
DisableDomain=yes
DomainList="x.com"
DefaultUser=xx.$TN
Password= encrypt=yes
MaxVNCD=1
VncPassword="" Encrypt=yes
VncPrompt=No

;*************************************************************
;* Services *
;*************************************************************

Service=SNMPD disable=yes
Service=ThinPrint disable=yes
Service=VDA disable=yes

;*************************************************************
;* General Session *
;*************************************************************

SessionConfig=ALL DisableSound=Yes EnableImprivataVC=no Fullscreen=yes DisableResetVM=yes
Reconnect=yes
UniSession=yes

;*************************************************************
;* ICA (for XenApp7) *
;* Change this section to point terminal to a different app *
;*************************************************************

IcaBrowsing=HTTP
Seamless=yes HideTaskBar=2
PnliteServer=x.com/Citrix/Cx/PNAgent/config.xml Storefront=no ReconnectAtLogon=2 ReconnectFromButton=2 AutoConnectList="x" Timeout=30
;PnliteServer=x.com/Citrix/x/PNAgent/config.xml,x.com/Citrix/x/PNAgent/config.xml Storefront=no ReconnectAtLogon=2 ReconnectFromButton=2 AutoConnectList="x" Timeout=30
;PnliteServer=x.com/Citrix/x/PNAgent/config.xml Storefront=no ReconnectAtLogon=2 ReconnectFromButton=2 AutoConnectList="x" Timeout=30
SessionConfig=ICA OnDesktop="x" EnableRTME=no Timeout=no

;*************************************************************
;* Troubleshooting *
;*************************************************************

;Privilege=High EnableTrace=yes
;SaveSysinfo=ftp://x/AISLENAME/wyse/wnos/trace/$tn.txt size=100