Fpmi/fsql/ms-sql

  1. Server MS Windows 2003, dual NIC cards.
  2. Managed 16 port ethernet switch.

  1. On server both NIC card go to Managed switch. One is addressed on the 10.x.x.x side, the other is addressed on the 192.x.x.x side
  2. FPMI, FSQL, MS-SQL are all on this server.
  3. All servers are on UPS.
  4. Managed switches are not on UPS.

Problem - Out of the past 6 weeks there has only been one monday morning when we have come in and our SCADA system has worked properly. The other 5 weeks each monday morning the system has been down due to loss of connectivity with the datasource.

If you go in thru the gateway and restart the PMI, the datasource will reconnect but as soon as you try to run any type of query it drops connection.


I.T. dept thinks that it is because the managed switch is not on a UPS and when their is a power outage the servers stay up and running because they are on a UPS but the managed switch does not and that in turn makes it a problem for FPMI and sometimes FSQL to log backin and reattach to the database.


Current config has both FSQL and FPMI looking at the IP address of the server. But I am wondering if since all these items are on the same server if looking back at local host would be a better option. Example: Instead of looking at say 192.168.1.80:5055, have it set at 127.0.0.1:5055. Just some thougths I am throwing out there.

=====================================

So has anyone had any similar problems, is this a known problem and all our managed switches should also be on a UPS, or any other thoughts greatly welcomed.

Hope this is clear and have a great day.

A few thoughts/questions:

  1. Do you have regular power outages? I doubt it’s the UPS…
  2. What is the exact problem when the SCADA system doesn’t “work properly?” Error messages or logs would be helpful here. I did see the comment about the dropped DB connection in FPMI.
    1a. What DB driver are you using? FPMI config page->Datasources->Drivers.
    1b. You’re using SQL Server 2008? What version?
    1c. What fixes it? A reboot?
  3. Has this only happened on Monday mornings? Did it work properly 6+ weeks ago? What has changed?
  4. Do you have separate VLANs set up on the managed switch? Is it uplinked/trunked to other switches? Describe what you’re doing with the 2 network address ranges.

During the past couple of weeks we have been having lots of storms pass thru the area. The one from this past weekend did the most known damage at this point. It dropped 3inch of rain and caused a large mud slide not far from our place of employment. There was however two weekend where the storms seemed relatively minor and in my opinion should not have caused power interuption, but something happened.

The driver we are using is Microsoft SQL JTDS1.2

I also noticed that in my previous posting I forgot the following data which may also help:

We are also running MySQL on the same PC with the native MySQL 3.1.6 driver and it reconnects just fine when the connection is dropped.

As for what has happened, well IT took the project over, they where using some kind of software called DeadLock Detector, and they were configuring it to email deadlock notices. Other than that they are somewhat tightlipped.

The SQL server we are running is not 2008. It is 2005.

As for what fixes the problem, you are indeed correct it is a re-boot that fixes the problem.

To the best of my knowledge it had been working quite fine. Of course we did not have the storms we had but it was working. And YES it has only been occuring on monday mornings.

I can only describe what I am doing with the addressing. All our production equipment are on the 10.x.x.x side. Our main operations and everything else is on the 192.x.x.x side. They also have seperate net masks. this way it keeps the production floor and the main seperate and it also keeps the PC’s that are used for this SCADA system on the production floor from accessing the web.

As for the other part of the question, I will pose it to the IT staff but dont know if you will get an answer.

If you can tell me what the command line in Internet Explorer would be to go to the logs on the server I would be more than happy to post those here. I have asked for a copy of the logs and have been told that since they think it is a device not being on the UPS problem that is all we are going to do and that is that. But I think there is more and I have seen Carl and Travis when doing a goto meeting type stuff in the address line and be able to view the log files in PMI.

Hope I have answered your questions clearly. Have a great day.

[quote=“nathan”]A few thoughts/questions:

  1. Do you have regular power outages? I doubt it’s the UPS…
  2. What is the exact problem when the SCADA system doesn’t “work properly?” Error messages or logs would be helpful here. I did see the comment about the dropped DB connection in FPMI.
    1a. What DB driver are you using? FPMI config page->Datasources->Drivers.
    1b. You’re using SQL Server 2008? What version?
    1c. What fixes it? A reboot?
  3. Has this only happened on Monday mornings? Did it work properly 6+ weeks ago? What has changed?
  4. Do you have separate VLANs set up on the managed switch? Is it uplinked/trunked to other switches? Describe what you’re doing with the 2 network address ranges.[/quote]

The logs are stored here on the PMI machine:

C:\Program Files\Inductive Automation\FactoryPMI\logs

or - Help->Diagnostics->Log Viewer in the designer.

more to follow…

I’ve had switched go flaky after power problems, but I think it’s unlikely, it’s happening on a regular basis.

My gut tells me that the deadlock detector isn’t working properly. After all, nothing can hose over a good computer like an IT dept. :laughing:

Just my opinion, which will buy you 1.5% of a soda.

My gut feeling (based on your other posts as well) is that there’s a problem with: SQL Server or the JDBC connection to it. Find log entries in FPMI or SQL Server that indicate the problem. Also monitor concurrent SQL connections and other such metrics - it sounds like your IT dept is working on this. There’s also an “official” MS SQL Server JDBC driver that you can try instead of jTDS.

microsoft.com/downloads/deta … 912601fdc9