On Call "Case Study"

I’ll call this a case study because that’s kind of what it looks like. There is no code (sorry - there isn’t much to see anyway), but rather a dialog of how we made it work. I hope this helps others build something, and gets some feedback to IA on how people are using it.

This on call scenario is for monitoring a remote pumping station that is critical in colder weather. It was recently upgraded to AB PLCs, integrated to Ignition and is connected via redundant fiber to where Ignition lives. I was very excited when Ignition came out with their voice notification module - it meant we could do away with an autodialer, and the numeric pager that got passed around (yes, pagers still exist). Myself and a local integrator spent about 3 days getting the alarming working how we wanted. I’m quite happy with the result.

Requirements

  1. Voice dial alarms (not all people on call have smart phones for email)
  2. Dial the person on call, dial the supervisor if they don’t acknowledge
  3. Always send the supervisor the alarm via email, and an email when it is acknowledged
  4. On call list changes weekly, but has no pattern
  5. Ability to change who is on call for a day or days easily, by potentially anyone

Considerations
We first looked at the built in rosters and schedules. If we used a single roster of all people on call and holiday schedules for those not on call, this mostly works. However, there is no option for odd schedules (they don’t repeat with any pattern), and swapping who is on call temporarily requires two changes and is therefore error prone. I also didn’t want to give people that much access to change them.

We then thought about having a roster that only had the on call person and the supervisor in it. Each week, they would update the roster, but again, this relies on human intervention. If there was the ability to edit the roster programmatically, we could have done this.

We looked at setting the alarm pipeline via a tag, and having different pipelines for each person, but there is no mechanism if an invalid pipeline is selected by some chance. This also necessitates a pipeline per person. Our solution avoids a pipeline per person, but not a roster per person.

The Solution
A screen was made with a calendar control that updated a database. The user can add an event and select a list of people on call, then give them a time period to be on call. There are checks to make sure two people cannot be on call at the same time. However, they can add an “alternate” to any time period which overrides the normal weekly schedule. Checks are done before the database is updated to make sure this is all consistent. The supervisor will typically fill this out for 2-3 months in advance.

A gateway script looks at the database at a timed interval and evaluates who should be on call at that moment, using the start/end times of the events. The event names are simply the name of the person. The resulting name of the person on call is written to a memory tag. If no one is on call, an alarm tag is written to and an email is sent to the supervisor.

The alarm tags that connect to the PLC have “associated data” added to the alarm condition, which is bound to the tag with the name of the person on call. When the alarm is triggered, this data gets put into the pipeline.

All alarms are directed to a main pipeline where they split to an email and phone pipeline (using jumps).

The email pipeline sends an immediate email the supervisor. Using an expression, it checks {isAcked} every 30 seconds for up to 30 minutes. If it is acknowledged, an additional email is sent so the supervisor knows the call was taken (unfortunately there is no way we could see to determine who acknowledged it). If it is not acked, we send an email saying it timed out so we don’t fill up the pipelines with junk (and they terminate after some time anyway).

Simultaneously, the voice pipeline feeds into a switch, looking at the associated data and determining who is on call. The outputs of the switch (one for each person) feed to notifications block for each person that notifies a roster containing just that person. If that fails, the supervisor is notified. If the switch doesn’t match a condition (someone screwed up the programming) the supervisor is called.

Lessons Learned (and features requested)
The pipeline editor is painfully slow with 15 notification blocks connected to a switch.

If we could programmatically edit the roster, we could avoid a lot of duplication on the rosters and notification blocks. In fact we could remove the associated data and the switch for the users.

If we could use the expression block to look at values outside the pipeline, we could avoid the associated data with each alarm. (ie look directly at the tag of who is on call). In our testing this didn’t work.

There is no way to set common dialing properties (dial 9 to get out), which made pulling the users from our active directly pointless. I created a database user source just for on call contacts so I could edit it in SQL if needed. At the end of this post, I just realized I could have used a generic database user and just changed their contact number on the fly…

On the positive, initially we weren’t sure if we could get it to work, but we found a way, so that’s a plus (yay Ignition).

Tom

1 Like

Tom, thanks for the detailed writeup, this sort of feedback is very useful.

I think the main thing that would help you would be a scriptable roster, where a script could return a list of contacts for any given set of alarm events. This is something that we have been talking about for a while and do still plan on doing.

Glad you found a way to accomplish what you needed!

Ignition Platform Version: 7.9.6

There are a bunch of threads about this same feature request, so I am just going to do it here.

I am looking down the barrel of having to schedule all users of my system in addition to scheduling machines as well. They all have repeating schedules, with breaks / lunches, and vacations / downtime. Two key requirements are that schedules can be viewed in the database and they can be managed through scripting. I need scripting access, because we might be able to integrate with ADP or another system down the road and pull people’s schedules directly without requiring management by our users. I also need scripting access because it will allow me to automate setup and reduce workload for users, say by assigning a schedule to all users in a department as they are added to the department or to a specific type of machine.

Is there any supported way to do this in Ignition?

If not, are there any examples or suggestions, so I can avoid common pitfalls?

For our complex scheduling tasks we’ve found that the iCalendar standard’s Recurrence Rule model is a great base. It’s both flexible and powerful, and there are already several implementations available in Python.

2 Likes

Interesting, would you mind expanding on that?
What libraries are you using?
How you can modify the calendar and store it?
What iCal schema variation - or do you store a flat file in the db?

Note: The icalendar and ics python libs don’t support interaction with a database
out of the box, so it looks like this will still be an uphill climb in the sense I will need to take an existing iCal schema format and build a package to work with it and modify a db backend.

Any problems you faced?

we designed a system similar to what you are looking for. we store the “schedules” in a database table and have scripting determine who would be on the rosters. The customer has 5 rosters and a night and day crew. They use screens that interact with the database to enter in schedules for each user like which shift they are working and what days. We then display the “schedule” using the equipment scheduling component(I think that is what it is called). we have a script that runs ever minute or so that processes the entries that the customer makes and marks them when the user should be on and then when they should be off.
Here is what the equipment scheduling component looks like. Then we just have basic screens to handle creating/editing/deleting any entries for the user schedules.

Then of course we have the alarm pipeline setup to query out the contents of the roster. each tag holds associated data of which roster it is assigned to and that data gets used in the script on the pipeline to determine which roster to query out.

This gives me some ideas, thank you.

Okay, so you must be using Calculated rosters, where you have a list of people to notify somewhere (presumably in the db), and then you pull the schedules for those people and narrow down the list to only the people on-call.


src: Ignition training video

That sounds pretty complicated, but based on responses so far this seems good.

If you or anyone else has a more fleshed-out idea or library suggestion to ease the pain, please reply; otherwise, it looks like a lot of custom work is ahead of me…

so the way we work it is as follows

we use a database user source where we can input users, just the run of the mill db user source and it is separate from security or logins, that way we can put anything in there without having to worry about active directory, etc.

then we have a table with roster names

then we have a roster entry table that holds the current users on all of the rosters

then we have a table for events where a user would input user, start date, end date, processed, completed, and shift

then we have a gateway script that runs every minute that looks at the events table and looks for any that have not been "processed " based on the start date. if it sees a new entry it will then modify the roster entry table based on who is scheduled to be on that roster and then sets a boolean column in the table that lets the scripting know that this entry has been processed. the script also looks at the end dates and will remove any users that should then be removed off the roster and then it sets a boolean in a “completed” column that lets the scripting know that this users entry has been completed and will need no further processing.

last but not least we have a script that runs once per hour that sends out an email to supervisors if nobody is currently scheduled.

Its not terribly complicated, it seemed like the toughest part was playing with the equipment schedule component to work as a people scheduler.

We use the dateutil.rrule implementation found here:
https://dateutil.readthedocs.io/en/stable/rrule.html

In the database we have a table for rrules/exrules which is linked to a table of rrulesets. The rrulesets are then referenced by the roster system and our system for executing scheduled events.

The tricky part was implementing a way to selectively update/expand the rrulesets into discrete calendar events. It's fairly simple to get all the rrulesets out of the database and iterate through the rrules to create a calendar, but you obviously do not want to have to rebuild the entire calendar when a single event is modified. By tracking the rrule that generated the calendar event you can rebuild the minimum number of events possible. Also, we implemented a caching system for far-future dates that only generates events when a user goes to view them.

Once the system was setup the process of adding rrules is pretty simple to handle.

If I was more Java-capable I think the system could make for a valuable 3rd party module.

1 Like