E_OUTOFMEMORY Errors on OPC Com Writes

I have Ignition 7.6.6 running with Java 7.0_60 in a redundant configuration with FactoryTalk Gateway OPC Server and sql server 2005 Enterprise edition.

Ignition is running on Windows 2003 Server R2 SP 2 on virtual machines.

Java Maximum memory is set to 1.5 GB.

Only sql bridge module is being used with OPC Com with various transactions both OPC Read and Many OPC Writes as well.

The same groups were running with the same OPC Server in factorysql for 3 years without any problems.

In Ignition, after every 5-6 days, I will get out of memory errors and the ignition gateway terminates and does not recover and has to be restarted.

Seems that OPC writes are the culprit here, I have it configured to Async already with 30 sec timeout. have updated java to recommended version already but to no avail.

Below are some messages and am attaching a portion of wrapper log as well. Attaching a longer wrapper also.

com.jniwrapper.win32.com.ComException: COM object method returns error code: 0x8007000E; E_OUTOFMEMORY (Not enough storage is available to complete this operation.)
com.jniwrapper.win32.com.impl.IUnknownImpl.invokeStandardVirtualMethod(SourceFile:758)
com.jniwrapper.win32.com.impl.IUnknownImpl.invokeStandardVirtualMethod(SourceFile:737)
com.inductiveautomation.opccom.interop.opcda.impl.IOPCAsyncIO2Impl.write(IOPCAsyncIO2Impl.java:60)
com.inductiveautomation.opccom.wrapper.subscriptions.COMSubscription$AsyncWriteJob.oleloop_startWrite(COMSubscription.java:1454)
com.inductiveautomation.opccom.wrapper.subscriptions.COMSubscription$WriteJob.internalRun(COMSubscription.java:1317)
com.inductiveautomation.opccom.wrapper.OleJob.run(OleJob.java:47)
com.jniwrapper.win32.MessageLoopThread$ThreadSynchronizedAction.run(MessageLoopThread.java:585)
com.jniwrapper.win32.MessageLoopThread$LoopThread.run(MessageLoopThread.java:527)
Error executing write to OPC-COM items.
Wrapperlog.txt (145 KB)

When this is happening, what is the Java.exe memory usage looking like in the Windows task Manager ? Is it capped or exceeding its memory limit ?

Java.exe memory in task manager is increasing every day by around 100 MB, so how within a 4 days, its already upto 913 MB and around 1 GB , Ignition gateway will stop and web page will not open even unless server is restarted.

This only started happening when Groups with OPC writes were enabled on the project and as you can see from the wrapper logs, the messages of our of memory are related to OPC writes as well.

I have configured Async Writes with 30 sec timeout. In the past 4 days , there have been 32619000 OPC writes, but the memory leak because of those is a problem because the same groups have been running in Factorysql with same OPC server software for a couple of years without any issue and the OPC server memory is stable.

Are you seeing any errors in the gateway console leading up to lockup in regards to the OPC write issues ? Also how many transaction groups are currently in the project and how many tags are in each group ?

Well it happened again and unfortunately, the backup system, which is set to warm, was also had the same issue and beoth failed when the master crashed. This happened on 7th August 19:17. Wrapper Log file is attached.

I have disabled async writes in OPC server properties since 7th August but still the memory is gradually increasing every day. I have currently 1200 Tags in 75 OPC groups, some at 1000 ms and others at 5000 ms. I disabled the 17500 tags which were in block reads in around 25 groups.

The wrapper logs are attached in which you should be able to see the messages just before crash.

Hi,

Can you get a thread dump (by going to Console>Threads in the gateway) when this happens again?

These types of errors are usually caused by blocked threads, though it can be very hard to tell what’s actually going on. Did you report something like this some time ago? I think I remember someone with FactoryTalk running into something similar, and perhaps deciding to stay with FactorySQL at the time.

Regards,

Unfortunately, when this occurs, the gateway web is not accessible so the thread dump is not possible at the time of the crash. which is why only the wrapper logs were sent. The attached thread dump is for right now when the memory is just high but no crash.

Yes the same problem was encountered with previous ignition versions as well when it was decided to stick with factorysql. The thing is factorytalk OPC server memory remains stable and the same OPC server was working with factorysql for many years without any issues.

Today the memory is already close to 1 GB and I can see that randomly the fsql-thread-1 to 5 are going into blocked or blocking some times and then they clear.
ThreadDump.txt (46.1 KB)