Project freezing, machine unstable

Ignition
1

Huge and complex situation there.

[code]#

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x721e0693, pid=3780, tid=6232

JRE version: Java™ SE Runtime Environment (7.0_60-b19) (build 1.7.0_60-b19)

Java VM: Java HotSpot™ Client VM (24.60-b09 mixed mode windows-x86 )

Problematic frame:

V [jvm.dll+0x140693]

Failed to write core dump.

If you would like to submit a bug report, please visit:

http://bugreport.sun.com/bugreport/crash.jsp

#[/code]

This happens just after the client jumps from the starting to the main project page. This one runs some scripts to load the project context.

After this error the gateway reboots, but the strangest thing is since that, the server machine resets (java exception: connection reset) every connection coming from the client IP, directed to any port (not only ignition port), and the even rebooting the machine, it continues to reset any connection :open_mouth:
On the other side, all the machine services are still available to other clients, coming from other IPs (of course, until they load the magic page).

I really don’t know how to study and solve this issue.
This came after some minor updates. The machine has very low CPU/RAM/DISK usage.
Attached the error pid file

2

Enlarging the gateway RAM and removing some bindings
(like “if(isAlarmActive(”/*"),“Please Check Anomalies”,“All Systems OK”)")
the problem seems solved :scratch:

3

again critical exceptions:

INFO | jvm 1 | 2014/06/30 09:35:49 | INFO [LocalDBManagerImpl ] [09:35:49,299]: Creating auto-backup of internal database "settings"... INFO | jvm 1 | 2014/06/30 09:35:49 | INFO [ENGINE ] [09:35:49,300]: checkpointClose start INFO | jvm 1 | 2014/06/30 09:35:49 | INFO [ENGINE ] [09:35:49,362]: dataFileCache commit start INFO | jvm 1 | 2014/06/30 09:35:49 | INFO [ENGINE ] [09:35:49,487]: checkpointClose end INFO | jvm 1 | 2014/06/30 09:35:49 | INFO [ENGINE ] [09:35:49,488]: Initiating backup of instance 'settings' INFO | jvm 1 | 2014/06/30 09:35:50 | 1 / 4 settings.properties... ERROR [LocalDBManagerImpl ] [09:35:49,946]: Cannot create autobackup for internal database "settings". INFO | jvm 1 | 2014/06/30 09:35:50 | java.sql.SQLException: java.lang.OutOfMemoryError INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.jdbc.Util.sqlException(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.jdbc.Util.sqlException(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.jdbc.JDBCStatement.fetchResult(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.jdbc.JDBCStatement.executeUpdate(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.apache.commons.dbcp.DelegatingStatement.executeUpdate(DelegatingStatement.java:228) INFO | jvm 1 | 2014/06/30 09:35:50 | at com.inductiveautomation.ignition.gateway.localdb.hsql.HSQLDBManager$AutoBackupDaemon.run(HSQLDBManager.java:618) INFO | jvm 1 | 2014/06/30 09:35:50 | at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:573) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.util.concurrent.FutureTask.runAndReset(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.lang.Thread.run(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | Caused by: org.hsqldb.HsqlException: java.lang.OutOfMemoryError INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.error.Error.error(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.result.Result.newErrorResult(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.StatementCommand.execute(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.Session.executeCompiledStatement(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.Session.executeDirectStatement(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.Session.execute(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | ... 12 more INFO | jvm 1 | 2014/06/30 09:35:50 | Caused by: java.lang.OutOfMemoryError INFO | jvm 1 | 2014/06/30 09:35:50 | at java.io.FileInputStream.readBytes(Native Method) INFO | jvm 1 | 2014/06/30 09:35:50 | at java.io.FileInputStream.read(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.lib.InputStreamWrapper.read(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.lib.InputStreamWrapper.read(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.lib.tar.TarGenerator$TarEntrySupplicant.write(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.lib.tar.TarGenerator.write(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.lib.tar.DbBackup.write(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.persist.Logger.backupInternal(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.persist.Logger.backup(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | at org.hsqldb.StatementCommand.getResult(Unknown Source) INFO | jvm 1 | 2014/06/30 09:35:50 | ... 16 more

4

Based on the logs you just posted, looks like still having memory issues on the gateway. Have you checked task manager to see what the Java.exe memory usage looks like? With message you are seeing it looks like the JVM for the gateway is running out of memory and that is preventing it from doing its normal backup.

5

java.exe is in the correct mem range.
java = about 700 MB
range = 1 GB min - 3 GB max

Some of the configured devices are down for some hours during the day, this is normal.
I have many offline and timeout messages in the console. Can i have any issue with this and the modbus2 driver?
Thank you

6

How often is it locking up and what are the hardware specifications for the Machine? In the logs it looks like you are running a 32 bit version of Java is this correct ? If so then this might explain what the project is freezing, because you have the JVM max set to 3 gb. With a 32 bit JVM its max size can only grow to 1.5 gb which is only half of the configured Max.

7

I’ve upgraded to Ignition 7.6, Java 8 and now i can set mem to 3 GB.
Thank you.