SENTRIS actuation web service (.asmx) returning wrong/stale results after a Server 2008 R2 glitch — actuations failed

[People & Places]

After the overnight Server 2008 R2 hiccup on the SENTRIS host, the actuation SOAP web service came back misbehaving: responses are slow, and &mdash; far worse &mdash; GetNodeStatus and PingSubject intermittently return the WRONG record (a stale RID, or an adjacent node's row). A handful of actuations were raised against a mis-resolved hardware address before the fault was spotted and the queue frozen. To reproduce, open the service test harness at <a href='/services/Actuation.asmx' target='_blank' style='color:#0000cc;'>/services/Actuation.asmx</a> and invoke GetNodeStatus or GetEstateSummary &mdash; the figures don't agree with the CMS. P1: this service sits in the actuation path.

[D. Pike]

First guess, and don't all groan at once: is it DNS? The service resolves the node registry by name and that box only just rebooted.

[M. Ross]

Not DNS this time, Pike. The .asmx worker came back on stale cached entitlements after the 2008 R2 restart and is reading a shadow copy of the node table &mdash; serving old data with a completely straight face. Queue frozen until it clears.

[S. Rao]

Confirmed against the CMS: GetNodeStatus on GAT-4471 returns GAT-4478's row. Anything actuated off this service in the last twelve hours needs checking by hand. Not a good one to have found the hard way.