Tag Quality/Device connection Problem

I am having a timeout problem with my RTU device over TCP.

Starting off with a few tags I have no issues of connecting or disconnecting. I get the data that I need and can communicate with the controller easily.

I then import ~3000 tags from a csv file. I have all the addresses from a communication manual and I know that most of them are valid.

Then - disaster. Once they are imported the device will timeout and then the setting that toggles “3 Timeouts requires reconnect” makes the device disconnect and reconnect. Once disconnected, all the tags go Null (even the few that were formerly working).

Now, I am lead to believe from other forum posts that this is a Timeout setting problem. I have tried all the proposed solutions and I dont seem to be getting any progress.

The current settings I have are :
Scan Class
Slow Rate : 1000
Stale Timeout : 10000
Mode : Direct
Data Mode : Subscribed

Device Properties
Communication Timeout : 2000
Force Reconnect : Enabled

This should be very close if not identical to the initial setup from a fresh install. Every other setting on the device is default.

The two errors that I am getting in the console log are as follows.

[code] drivers.ModbusOverTCPDriver[Controller RTU_TCP].ReadHoldingRegistersRequest Request failed. FailureType==TIMEOUT

java.lang.Exception: Request failed by TimeoutDaemon due to timeout: ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@fc3a58]
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.failRequests(BasicRequestCycle.java:349)
at com.inductiveautomation.xopc.driver.api.BasicRequestCycle$TimeoutDaemon.run(BasicRequestCycle.java:305)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

WARN 1:30:29 PM drivers.ModbusOverTCPDriver[Controller RTU_TCP].BasicRequestCycle.TimeoutDaemon ScheduledRequest[com.inductiveautomation.xopc.drivers.modbus2.requests.ReadHoldingRegistersRequest@fc3a58] request with key “java.lang.Object@1ac81c9” failed due to timeout.[/code]

Things I have tried:
Turn Device Connection Timeout to 30000 - results in Data never getting to Tags.

Turn Device Connection Timeout to 1000 - results in device being stable for a few seconds, data that is pushed to tags are incorrect values. Once disconnected stays that way longer.

Turn Scan class slow rate to 500 - Data is cyclically obtained through the list of tags but values are incorrect and disconnects after every 3 sets.

Turn Scan class time Stale timeout to 15000 - No noticeable effect.

When disabling forced reconnect after 3 timeouts - all of the tags become bad quality.

Each one of these changes have the same error in the console log.

If all tags are disabled Connection timeout is set to 2000 slow rate is set to 1000 and stale timeout is set to 10000 - I can get data without errors for 5 tags. I then slowly increased the number of tags active (5, 60, 150) These all received correct data in a timely manner as well as kept the controller connected. Once I added up to 345 tags the disconnecting and bad quality occured again. Slowly removing these tags all the way back down to 5 active tags did not have any affect and the tags remained bad quality.

Try setting your scan class slow rate to something like 10 seconds.

Use a comm. timeout on the driver of maybe 5 seconds.

Disabling forced reconnect will prevent the null values when the device disconnects, but you’ll get stale/bad quality on your values once enough time has elapsed without them being updated.

Doing this gives the same error in the console except every 5 seconds.

Data in previously Good quality tags is not correct but still shows good quality.

Data in tags previously turned off shows as Unknown quality with a value of null.

The incorrect data is curious but… timeouts are timeouts. If the device doesn’t respond in time there’s not a lot we can do from the driver side other than continue to slow things down more and more.

If you go the diagnostics page for that device (diagnostics link next to the device entry) does it tell you how many items are subscribed and how many requests are representing those items?

I believe this is what you are referring too

SubscriptionItem Count 3717
ScheduledRequest Count 53

Are there any settings on the modbus TCP gateway you can play with? Anything about timeouts?

I have two options for the gateway - they both seem to share a version of this problem.

The first is a TCP gateway that has a timeout selection. This gateway’s problem is that as more tags are inserted it starts recieving the wrong data. When I send a value through the gateway the controller recieves it, it then proceeds to change to a seemingly random number after that. (Modbus TCP)

The second is a hack job of sorts - its a serial to USB adapter with a simulated COM port - with a virtual COM port to TCP port - not a huge shocker this isn’t working very smoothly. The adapter does have time out settings but they are all managed with the Computer’s COM port settings. The virtual COM ->TCP has options for time out settings. I believe that this is where the bottle neck is happening though, but I dont know of a way to measure this over this many conversions. (This is also using Modbus RTU over TCP drivers)

I am thinking that the requests are flooding the controller regardless of which gateway is being used. Is there a way to stagger (Mostly the initial) as well as the rest of the queries (I have been trying Scan Classes but with no luck) I thought possibly a gateway script that would update X amount of tags each Y amount of time - is there a way to enable and disable tags via a client script the help file seems to be a short on system.tag information.

I attached a picture of the queue times which are ridiculous at around 20 seconds as well as duration. (With TCP gateway)

Discovered that Ignition default settings for Max Holding Register Read Requests of 125 was completely swamping every part of the system and made data un-retrievable.

Just curious what you turned the rate down to?

Pssst! That user hasn’t even visited here in eight years. Don’t hold your breath waiting for a reply.

I thought I’d give it a try anyways Maybe Ill make a new post…