Data Loss due to INACTIVITY?

I am using KEPwar Enhanced OPC/DDE Server V4.300.449.0 - U
Drivers: AB Suite: Allen-Bradley DH+ V4.30.50.0 - U
: Allen-Bradley Ethernet V4.70.79.0 - U

1. Can you verify that: a. FactorySQL is set up as a Windows service (control panel->admin tools->services) b. The service starts automatically ("startup type" set to "automatic") c. Data logs automatically without the frontend/system tray application open (reboot without a user logging into the FactorySQL computer and check the database)

  1. It was STARTED and set to MANUAL. I have changed it to AUTOMATIC. I rebooted and nothing started working until I logged in. Also, we have NOVELL installed on this computer and the login prompt is NOVELL’s not windows. Not sure if this effects anything.

Queston on the SERVICES, when you double click on FPMI or FSQL or MySQL and select the “Log On” tab, it is set to “Local System Account” and the “Allow service to interact with desktop” check box is not checked. Is this correct? And what settings should I use on the “Recovery” tab?

2. The system tray application can provide the same functionality as the service to support legacy OPC server applications. I wouldn't be surprised if the user logging on at 3:01 caused the system tray app (not the frontend) to start logging data.

So, the “USER” being on ANOTHER seperate computer, connecting to the server, could cause the system tray app to start? Also, I miss spoke. At 3:01 the user clicked on a BUTTON withing the client and accessed a Graph. So, the user was already logged in, but I’m not sure WHEN they logged in, either BEFORE it stopped logging, or WHILE it stopped logging. But in the past, I have come in on a MONDAY and logged in and everything is working fine till I look at a trend and noticed that there is no data for lets say SUNDAY, but it started logging the exact moment I logged into the client.

Just checking the obvious but what kind of power options does the computer running FSQL have set? Is it set to power down the hard drive or to “standby” after a period of “non-use”? User activity may be “waking-up” the computer.

Check the user account that’s set to run the service (admin tools->services->FactorySQL service properties window, click the “log on” tab).

I don’t have much Novell experience, but I suspect that a local account should be fine to run applications, even on a machine that’s a Netware client. You’ll want to do everything you can to isolate/verify the fact that the service is or is not starting on it’s own.

  1. It can’t be power/inactivity settings because you’ve verified that the system doesn’t start automatically on a reboot.

  2. Trying to connect a remote frontend can’t start the service on that distant end. You can start and stop groups remotely, but the service/tray app had to have been running already.

Normally, the Novell Client takes precedence over everything else. It’s why none of the services start until after you log in.

However, there’s a utility in the Novell Cool Solutions pages called Autolog that lets you automatically log in a workstation. One of the options you can set up is to login to the local workstation without logging in to Novell (I have a training station set up this way). This should take care of the services problem without compromising the Novell security.

8)

Hope this helps!

Now, instead of losing data from KEP or FSQL issues, I am having issues with the gateway.

It will run fine for a week or so, but when I come in at 7am, the gateway will not be running.
So, naturally I start it back up and goto the "System Console" on the gateway interface.

Here is the ONLY thing that is on it ONCE I have restarted it.

And in Window's Event Viewer I have this:

It appears that the TIME at which I got THIS error, is VERY close to the time at which I tried to log in this morning.

Any ideas??

Another observation. A user was ABLE to log in at 7:00 am with no problems, BUT when he logged in, all of our SQLTags were stale and stayed stale. Also, every screen he went to, had an xml parsed error.

Is Kepware set to start as a service? I didnt see any mentioning of this

This is now not a data loss issue, but seems related to IA software. This is an issue just with the GATEWAY. Data is still being collected, just no access to FPMI till I restarted the gateway.

Yes, the messages indicating that the service has stopped unexpectedly are not good- either there’s an error occurring inside the gateway, or some outside entity is force-quitting the gateway process.

I’ve never seen the gateway application crash completely- so I’m inclined to look towards the second possibility.

What was the solution on the FactorySQL side? Is there any chance that the Novell stuff is interacting with the FactoryPMI gateway application?

However, we should probably also start by having a look at the PMI gateway logs. Could you go to “logs” under the FactoryPMI install directory, zip up a few of them that are around the days/times that this problem has occurred, and email them to support (at) inductiveautomation.com?

Regards,

On the FSQL issue, I was able to set as service, but the AUTOLOGIN for novell didn’t seem to work too well. LOL We have NOT had any other issues with that though.

As far as the FPMI gateway is concerned, here is what I have:

<tr>
<td>Oct 14, 2009 07:26:52 AM</td>
<td title="Level"><font color="#993300"><strong>ERROR</strong></font></td>
<td title="Message">Servlet.service() for servlet Gateway threw exception</td>
<td title="org.apache.catalina.core.ContainerBase.[mainEngine].[localhost].[/gateway].[Gateway] category">org.apache.catalina.core.ContainerBase.[mainEngine].[localhost].[/gateway].[Gateway]</td>
<td title="http-80-Processor10 thread">http-80-Processor10</td>
</tr>
<tr><td bgcolor="#993300" style="color:White; font-size : xx-small;" colspan="6">java.lang.OutOfMemoryError
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at java.util.zip.Deflater.init(Native Method)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at java.util.zip.Deflater.&lt;init&gt;(Unknown Source)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at java.util.zip.GZIPOutputStream.&lt;init&gt;(Unknown Source)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at java.util.zip.GZIPOutputStream.&lt;init&gt;(Unknown Source)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at com.inductiveautomation.factorypmi.gateway.sqltags.db.DatasourceTagProvider$Base64GZippedPrintWriter.createXMLPrintWriter(DatasourceTagProvider.java:4715)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at com.inductiveautomation.factorypmi.gateway.sqltags.db.DatasourceTagProvider.processClientPoll(DatasourceTagProvider.java:880)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.runSQLTagPoll(Gateway.java:1153)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.doPost(Gateway.java:407)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at com.inductiveautomation.factorypmi.gateway.ErrorReportValve.invoke(ErrorReportValve.java:95)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
<br>&nbsp;&nbsp;&nbsp;&nbsp;	at java.lang.Thread.run(Unknown Source)
</td></tr>

There are many MANY threads like this. There are anywhere from 4 to 10 of these errors PER/SECOND. It start to give these errors, then about 25-30 minutes later, that is when it crashes.

Occasionally I get this one:

ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error 
     at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366) 
     at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433) 
     at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348) 
     at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:392) 
     at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381) 
     at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:88) 
     at com.inductiveautomation.factorypmi.gateway.servlets.Gateway$ContentLengthBufferingOutputStream.write(Gateway.java:313) 
     at java.io.OutputStream.write(Unknown Source) 
     at com.inductiveautomation.factorypmi.common.utils.Base64$OutputStream.write(Base64.java:1491) 
     at com.inductiveautomation.factorypmi.common.utils.Base64$OutputStream.write(Base64.java:1539) 
     at java.io.ByteArrayOutputStream.writeTo(Unknown Source) 
     at com.inductiveautomation.factorypmi.gateway.servlets.Gateway$2.run(Gateway.java:2201) 
     at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.doDBAction(Gateway.java:2364) 
     at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.runQuery(Gateway.java:1925) 
     at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.doPost(Gateway.java:404) 
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:709) 
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) 
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) 
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) 
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) 
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) 
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) 
     at com.inductiveautomation.factorypmi.gateway.ErrorReportValve.invoke(ErrorReportValve.java:95) 
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107) 
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148) 
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869) 
     at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664) 
     at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527) 
     at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80) 
     at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) 
     at java.lang.Thread.run(Unknown Source) 
    Caused by: java.net.SocketException: Software caused connection abort: socket write error 
     at java.net.SocketOutputStream.socketWrite0(Native Method) 
     at java.net.SocketOutputStream.socketWrite(Unknown Source) 
     at java.net.SocketOutputStream.write(Unknown Source) 
     at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:746) 
     at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433) 
     at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348) 
     at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:769) 
     at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:125) 
     at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:579) 
     at org.apache.coyote.Response.doWrite(Response.java:559) 
     at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361) 
     ... 30 more 

Any thoughts on this??

That first error shows that the Gateway has run out of memory. Stack traces and log messages are pretty useless for debugging this.

How is the memory looking for the Gateway now (I assume you’re restarted it)? You can check on the Gateway Status page, and also report what Windows thinks the Gateway is using through the Task Manager (FPMIGateway.exe)

Gateway Status page says Memory (used/max) = 166.85mb / 1016.12mb
Taskman shows 250,000 K (aprx).
Taskman shows 524,144 K for MYSQL.

I’m still having this issue, any ideas? (gateway)

Are the logs the same, with out of memory errors, etc.?

yes they are… I could send you the rather LARGE log file LOL about 38 megs

Have you tried zipping it? We’ve found that the log format zips up really well. And then, instead of emailing it, you could go to inductiveautomation.com / upload (without spaces) and send it to us.

Regards,

This happened again today. Gateway crashed Jan 20 11:02:37 a.m. I stopped and restarted thegateway around 11:14 a.m. Attatched is the log file for today.
[attachment=0]FactoryPMI Gateway Log Jan 20.zip[/attachment]

Bump

Sorry for the delay - sometimes I miss threads that are buried in unrelated topics.

That log shows that you got an out of memory error while saving your project. How big is this project? Go to the “view” icon in the Gateway > Config > Projects > Manage and see how large it is.

After that it appears that the database stopped servicing queries, because your db connection pool was full continuously, although my confidence in that diagnosis isn’t very high. But thats what the logs are telling me, FWIW.

Can you give an overview of the stats of this machine/system. where are the pieces running, what are the server specs, how many clients connected, how many quries per second, etc.