Eurotech will showcase the benefits of running Ignition on an ISA62443-4-2 certified device. This demonstration will highlight how Eurotech's advanced device management capabilities can simplify the process for OT systems integrators to securely manage applications remotely. Attendees will gain insights into how the integration of Eurotech's ReliaCOR 40-13 Industrial PC with Ignition software provides a robust and cybersecure foundation for industrial applications. This collaboration not only meets stringent cybersecurity standards but also enhances the efficiency and scalability.

Transcript:

00:06
David Bader: My name is David Bader. I lead business development for a company called Eurotech. Has anybody heard of Eurotech before? A few? Yep. So I've been with Eurotech for about a year. There's, I'm the least I have the least longevity with Eurotech, even for companies that are not Eurotech. There's a lot of Eurotech employees and other companies here as well. Dave.

00:21
David Woodard: So I'm David Woodard. I'm a Solutions Architect with Eurotech. I've been here a bit longer. I've been with the company for 11 years and have been in IoT and industrial automation for probably closer to 15. So pleasure to be here.

00:36
David Bader: Yeah, I beat him out in the longevity of being in the business for sure. So I've been doing automation for 40 plus years now. So I've been involved in systems integration and distribution. I worked for AWS for a short time and led robotics for AWS for a while. The idea of coming to Eurotech was to be able to bring that kind of security level that AWS and the other cloud providers offer in the cloud down to the edge. So Eurotech is, if there's one thing that you take from this, it's that Eurotech is a company that provides enablement at the edge, right? So we provide a secure way to orchestrate and maintain your systems kind of at the edge. And we're gonna talk a lot about cybersecurity today. There's two themes, overall themes, that we that we kind of talk about from Eurotech. It's cybersecurity and the ability for systems integrators and OT providers to be able to uplift your cybersecurity posture in an area where we normally would turn that over to IT, right? So the concept is if we can provide the IT level of cybersecurity down to the OT space, that's kind of gonna be the theme that we're gonna talk a little bit about.

01:55
David Bader: And then the other one is to maintain kind of a secure remote access and remote device connectivity, right? So being able to do things that you normally would do by plugging into the device remotely, but in a secure manner, right? So being able to do VPNs that have the IT kind of functionality that you would expect from an IT perspective. So I'm pretty informal. If people have something that's genuinely on your mind, say it, but we are gonna have some questions and answers at the end. So I'm gonna talk a little bit about what Eurotech does and where we are from a... why cybersecurity is super important. And I'm gonna it over to Dave at the end, that he's going to talk about this brand new cybersecurity wizard that we're introducing here for the very first time. So you guys, the very first time you're, hearing anybody's hearing about this. Dave's going to do a demo on that.

02:49
David Bader: So Eurotech's been around for 30 plus years. We're headquartered in a small town in Amaro, Italy, which is in the Northeast part of Italy, all the way up near...

03:00
David Woodard: Austria.

03:00
David Bader: Austria. Thank you.

03:01
David Woodard: Never think of it.

03:05
David Bader: We have operations in the U.S., we have a bunch of people in the U.S., we have people in Canada and all. So some of the things that we've kind of been known for over time way back in the beginning of the, of the Eurotech history, we kind of worked with some people people that worked for Eurotech at the time developed a small protocol called MQTT. Anybody heard of MQTT, right? Yeah. There's a few people that certainly know MQTT, right. So over time we've kind of evolved from kind of a board manufacturer into a industrial automation solution provider for hardware and software. So we're very excited to now be part of the Alliance Program and out in a table 11 right across the hall here is the very first Ignition Edge, a piece of hardware that is certified to IEC 62443 IEC, ISA 62443 cybersecurity standards. And I'm gonna talk a little bit about what that means to get through some of this stuff. Our portfolio is pretty big, right? We build hardware from gateways for different applications for transportation, for industrial automation, for medical, all the way up to pretty beefy GPU-based processors that run AI and those kinds of things.

04:25
David Bader: So for any kind of application, including running Ignition on any of these devices almost, we can meet your needs. But again, the differentiator is that cybersecurity and that remote device access in a secure manner, right? So I would guess, right? I've asked a few questions already. How many people talk about cybersecurity with their customers on a regular basis? So a good portion, which I would guess, right? Because you're in this room and you wanna learn a little bit more about it. What I find really interesting is there's also an equal number of people that don't talk to customers about cybersecurity, right? They say, "Hey, that's an IT function." And I think we've passed that threshold in the space where we have to talk from an OT perspective about cybersecurity because there's a large percentage of cybersecurity efforts that are being, that are stemming from the OT space. So is that something that you guys, that resonates right from an OT space? 20% of all of the attacks are happening from an OT, from the factory floor, which when I put this deck together, I was knocked out by that number. I would have thought it was 2% or 3%. It's more than 20% now. It's pretty amazing.

05:41
David Bader: So when you think about that, what does that turn? What does that kind of mean in terms of dollars? It's significant, right? So the average financial impact from data breaches way back in 2018 was seven and a half million dollars. It's significantly more now. But why, right? The concept is we have more connectivity in the factory floor now. It's not relevant that the person on the floor, he's not missing anything. He's just not been trained necessarily in cybersecurity. It goes back to that conversation earlier where I said, most people aren't even talking about it. We've got PLC systems. They're out on the floor. Anybody got GE 90-30 back still running in their plants. Right. So these things were not designed with cybersecurity in mind. Right. So and now we're asking to put more SCADA, more capability, more MES in the plant floor that are opening up all kinds of vulnerabilities, if we don't think about it. Yeah, I messed that one up. So this is just a brief slide that you can't read, right?

06:46
David Bader: That's too small. But it's a timeline of things that have happened in the world in the last 10 years or so, right? Maybe 20 years. So Stuxnet, that resonates. Everybody's heard of Stuxnet, right? That was a, that was a cyber attack that came through an HMI, right. On a machining center. So all the way up through Target got hit, Jeep got hit, BMW got hit, the Ukraine power grid, right? That's a big deal. These things are significant impacts to the world. So I picked two, two unique applications that happened in the OT space, Brunswick Corporation, billion dollar company that makes boats. They got attacked in June of '23. So that's pretty relevant, right? Really close, only a year away. And it disrupted their entire facility and it cost them 85 million dollars, right? So that's just a small company. Well, it's a big company, but it took their quarter two financials. They saw a big impact in their quarter two financials. So if I'm a CEO, I'm pretty upset about the fact that somebody was able to breach my system through a, through the factory floor. Right. And then on another level, there's a company called Applied Materials, which are big company, right? Multi-billion dollar worldwide company. February '23, still relevant.

08:09
David Bader: They got hit through one of their suppliers. So one of their major suppliers was attacked. There was a vector that came through. They were, they had vulnerable, they had access to Applied Material's systems, and it, and the attack came through their vendor. And that one cost them 250 million dollars. So we're not talking about peanuts when we're talking about OT attacks, right? This is significant dollars and significant impact to the business. And if you're a systems integrator and you're not talking about cybersecurity, in my opinion, this is a line of, sorry, in my opinion, this is a line of business, right? This is a way for integrators to spin up a new, a new way, a new piece of business, right? Talk to your customers about cybersecurity and how you can elevate it. So I'm gonna go fast. We build, everything that we do, everything that we're talking about today is about certified cybersecurity. There's a lot of ways that people address cybersecurity in the OT space.

09:08
David Bader: We think that being certified, building to standards, designing from secure by design is a significant piece, right? What we do is we have the ability from a remote access perspective to use VPNs. Everybody probably has the way to use a VPN, but we have an on-demand VPN capability that allows for automatic teardown, right? So if you're an IT guy, automatic teardown makes a big difference. Being able to connect, remotely access, do what you need to do, and then have the VPN shut down automatically. So you don't inadvertently leave it open and leave that attack vector space available. And please, Dave, jump in if there's anything that... So nobody wants to be this guy, right? Like it's not a good thing. So secure by design, right? The whole idea is if we do this right, everybody benefits, right? Suppliers of the hardware, suppliers the customer, and then we maintain that kind of security from the start, right? So again, I'm talking about secure by design. You have to build it from the beginning. It's not something necessarily that you walk in and say, yeah, I'm gonna create this really high level of cybersecurity in your plant without looking at the overall architecture.

10:31
David Bader: So you do a risk assessment, right? We don't do that. That's not what we do. I mentioned before, we provide enablement into the mitigation process. But you reach out to companies that provide cybersecurity risk assessments. They tell you where the vulnerabilities are. If you guys have that capability, that's fantastic. I think that that's a good way to do it. And then you build, you build these, you buy these features and these capabilities into the solutions that you have. So we provide hardware for running Ignition, right? But wouldn't it be great if you could buy the hardware that runs Ignition that also has this high level of cybersecurity and gives you this remote, this secure remote access capability. And that's the method that we are, that we're talking about. So we're doing it at 62443-4-2-1. So you're not gonna remember that specification. Excuse me, -4-2 as service level two, right? So Inductive Automation is already certified to 4-1. We're certified with our hardware to 4-2. And then the customer then can quickly and easily certify their entire system to 4-3. And that's really the enablement that we're offering is being able to have that customer get to a certified cyber solution in the field very quickly.

11:49
David Bader: And if we, if in the past, what you'd have to do is buy a piece of hardware that was hardened to a certain level. And it limited some of the functionality that you were able to load onto a server, onto gateway or other hardware. Right? So now with this wizard that we're going to talk about in a minute, is you can make these decisions in the field and work with the IT department to say, we wanna certify, or we wanna harden to this level. We wanna harden to 62443 right? Or we don't need to harden to that level, but we're gonna, we're not just going to leave everything open. And we'll show you, we'll walk you through a video that shows that, how that works. So what does secure by design mean, right? You wanna follow zero trust kind of principles and they're very standard and very well defined. So being able to say, we trust no one and nothing, right? So if you start to pass along keys and email certificates and all of that stuff, all of a sudden that becomes a real problem, right? That's not secure. If you're, if you have to email someone a certificate that's standing in front of a machine, inherently that's gonna be a problem because who knows who could get to his email, right?

13:03
David Bader: And then a continuous auditing and monitoring. So when we talk about zero trust, we talk about it from an entire ecosystem perspective. So we manage the certificates, the security certificates, from the TPM level, from the chip that's on our device all the way through to the IoT devices that are connected. We manage all of that. We maintain them. We keep them current and you don't have to worry about that as an integrator or as a customer or in any way. So I think that that's a really important piece. This slide, if anything you get from this presentation, the fact that Eurotech does this for you in an IoT perspective, and then also can do it all the way to the cloud. If your application calls for connecting to the cloud, that's super important, right? How am I doing? Okay. So I talk a little bit about 62443. I mentioned that maybe some have heard about it. There's a lot of things and you see here that I'm, talking about ISA and IEC 62443. Excuse me.

14:20
David Bader: So the, one of the key things is if you build to a standard, it's no longer subjective, right? So Eurotech many years ago decided that this standard was going to be kind of the worldwide, kind of the bar in which, which people should meet. Turns out that we were right. We made a good bet. It took us about two and a half years to become IEC 62443-4-2 SL2 compliant. And now we are the first and only, quite frankly, company that builds IoT hardware and enablement to that level. There's a lot of people that build to those standards, but have not yet gotten certified, right? We don't think we'll be the only ones. We think that we were the first, which is good to be first. So we use independent testing to validate that we're built to those standards. Again, what that does is it allows you to talk to your customers about building a secure system and maybe your customer doesn't want to certify. Maybe they don't wanna get to a 4-3, but you can say to them, "Hey, these are all of the components that you would need to, if you wanted to get to certify to a standard." Now, if anybody's from Europe, anybody here from Europe?

15:39
David Bader: Yeah, there's a few, right? It's not a guessing game anymore. It's required, right? You, you have to develop, you have to deliver 62443 standard products just to meet the law, just to meet the requirements. So we're an Italian company. We build in Italy and in Germany, all over the world, quite frankly, but we know that this standard is going to permeate not just in Europe, but beyond Europe, right? So what does that mean to us in the U.S. or in Canada? It's not mandated. It's not something that they're saying that we have to do, but quite frankly, it's an ROI conversation, right? It's something that when we talk to customers about this, we can put dollars and cents. I just showed you 250 million dollars right? It's pretty hard not to show the ROI on an investment in a piece of software that has a little bit more cost to it to get to that standard. But it's helping to prevent that 250-million-dollar hit, right?

16:37
David Bader: So even if the U.S. isn't mandating it, although we do mandate cybersecurity now in a lot of ways, right? It's suggested in a lot of ways. I think this ROI discussion in this line of business discussion for the integrators in the room is super important, right? We can now talk to customers about a higher level of cybersecurity at their OT level, at their OT floor. Make sense? How am I doing?

17:00
David Woodard: Fantastic.

17:05
David Bader: Okay, good. I like constant feedback. How am I doing? You guys feel pretty good so far? Okay. Nobody's left the room yet, which is very unusual actually. So I mentioned about how long it's taken and we like to show this slide on like every presentation that we do because it's actually a physical document that you get when you get to certification, right? It's not, oh I built to this standard, but I didn't get certified. No, we've actually gone through the certification. It is a physical document that we can send to you and, and say to your, to the IT team, look, we're buying product that is built to these standards. So how does this resonate worldwide? Right? There's a bunch of teams, people from Europe here. Obviously we talked about that. And then in vertical industries, right? In vertical industries, the 62443 standard it kind of travels to different areas, right?

18:00
David Bader: So if you're in industrial automation it's 62443, if you're in rail, it's Shift2Rail, energy is 62351, and so on and so on. Right. So there's, TSA is involved. So there's a lot of different almost, every standard is actually adopting 62443 as the core to the standard and then put it, putting it into individual, their individual requirements for their particular vertical industry. So I would say that in this slide, there, we'd be hard pressed not to touch every person in this room at some point in one of these verticals, right? Everybody's touching something in these verticals, right? And if we can meet the 62443 requirement, then these are all reciprocal standards that view 62443 as a, as kind of a guide, right? So if you've got a customer that's in a process automation and they're saying we need to meet TR 63069, then we can go in and have a conversation about 62443 and how that is actually 63069 at the core.

19:10
David Bader: For medical, I think medical is... Medical up there, we have medical 60601, 100% copy and pasted from the 62443 standard. So if you're in the medical space and customers are saying, "Ah, you gotta build something to 60601 standard," we can do it. We can help you. Make sense? Okay. So then I mentioned that we're certified to SL2. What does that mean, right? So I thought it was important to kind of make sure that people understood what that means. So the idea, right, is the SL1 is the components, right? To protect the components from casual access, casual mistakes and things like that. One of the things that the standard actually does is also includes tamper resistance, right? So if somebody goes in and messes with the server, there's a switch inside the server or inside the gateway, that is a bit, that ties back to our software that you can enunciate in Ignition or send an email from our software or any of those kinds of things.

20:08
David Bader: So if somebody inadvertently, a maintenance guy comes in and says, "Hey, I gotta upgrade the firmware or something on this," they can immediately get a response. You can literally shut the computer down if it's an onsite breach. So there's lots of ways that you can use that tamper resistance piece. And then SL2 is actually designed to mitigate and kind of prevent generally acceptable or generally recognized attack vectors. So Eurotech again felt that it was important enough for us to get certified to the SL2 standard. Not too many people have considered that. All the standards that we meet, not all of them, but many of them. So today, I thought that it was important that we talk about how do you get there, right? Like how do you put a... Take a computer, put it on a shop floor, what do you have to do to get to maintain that 62443?

21:08
David Bader: And these are all the steps. I'm not gonna go through every one of them, but there's at least 10, maybe more, steps that you have to do to build and harden an industrial IoT device to this standard. So what we've done is we've said, "Okay, you know, let's build something. What are these capabilities, right? So what are the advantages of having this?" There's a lot of words here, but the bottom line is that it's maintaining and monitoring to a rigorous standard the integrity of the environment, right? And then, can I ensure that it maintains that? So, yes. Right? So the idea is that when you certify, all of this gets continuously updated and as you keep your hardware and software current, it gets updated. So again, I mentioned this wizard that we're... Dave's gonna do a demo on.

22:00
David Bader: But the idea is, how does this work during deployment, right? So you can load all the software, whether it's Ignition or other software that you want. Then you walk through this wizard and it guides you to the level of security that you wanna provide at this OT space. Unheard of. Literally takes all of those steps that we talked about before. Excuse me. Now we can do it in the field and then can we be... Can we maintain that security with Ignition? Make sense? Lot of words, but pretty important. So Dave's gonna go through the video, he's gonna talk to that, and then we're gonna do questions and answers. So we went pretty quickly. Hopefully this touched a little bit. It wasn't just commercial. It was about providing some relevance to the market and where secure by design and standards really matter.

23:00
David Woodard: Great. So now that Dave's finished, we can come back to reality of doing all this, right? 'Cause if you wanna do this for a new customer or existing customer, doing all that level of security is really difficult. I think it's one of the most challenging parts of what we do, 'cause we have to understand the IT side and the OT side and how to do like, you know, understanding like that bridge is incredibly complicated and is very hard to do well. So that's why we came up with this, right? So, 'cause what I see a lot when I do integrations or when I do deployments or installations, is you don't do it. You say, "Okay, we need to get the POC working. We need to get this application working. We'll do security when once they buy in." Right? So once they say, "Yes, we wanna do it," then we do security. And then you realize that security is breaking what you did. So the wizard, what it will do for you, I'm just gonna play this video.

23:49
David Woodard: And I'll just talk while it's playing. So basically it's just a web application that's running on these gateways. So all of our gateways from the edge devices up to our more server class boxes, provides you with this walkthrough interface of setting up networking, enabling the secure elements that you need, and being able to do it while you're doing the deployment, right? Or if you say, "Okay, we just wanna get it working, but then we wanna see what happens if we enable this SSH policy. We wanna see what happens if we try to do this other thing." You can come back to this wizard, enable that feature, and see if it still works. So there's nobody on the command line, there's nobody like hacking your Linux file system stuff. There's nobody doing that crazy stuff in a working factory or in a pilot.

24:38
David Woodard: You do it in this wizard. Let the... Let our software manage it. And so here you can see it's doing the... All these things are relevant for the certification Dave was mentioning. So if you want to come here and just say, "Hey, I just wanna be IEC certified," you can click one button. It enables all those features and you're done. This video I think is three or four minutes, but you can do it in less than a minute. And I think even more importantly than that is this is not even necessary, right? So once you do this, once you say, okay, this is the standard that we want, these are the settings that we need, this is just a configuration for us, right? So you can say, okay, now I need to order 10 of these boxes, or a hundred, or a thousand. They can come preinstalled with all these settings already on there.

25:19
David Woodard: You don't have to worry about it anymore. I think the other cool thing about this, so you'll see now they're actually going through some provisioning with AWS. We can do the same thing for Azure. We can do the same thing for custom cloud endpoints. It is an extensible interface, right? So if you say, "Oh, but you know, we need this custom thing for our cloud services or for our customer," it is an extensible platform that you can add on to. So think that's it. We do have it running on this box I have here in front of me. So if you wanna come by our booth, I can plug this in and show you live what it does. We didn't wanna do that here just for timing, but, I'm happy to show that to you if you wanna come by the booth. So I think leave time for questions.

26:01
David Bader: Yeah. We have plenty of time for questions actually. Yeah. I think we have a half an hour for questions.

26:06
David Woodard: No.

26:08
David Bader: I'm just kidding.

26:10
David Woodard: Okay.

26:10
David Bader: Yeah. I'm making the guys in the booth nervous. Go ahead. Yes, sir.

26:15
Audience Member 1: So you are saying you're providing a software stack only for the remote device, the edge device, or there is also a kind of some cloud platform?

26:23
David Woodard: It's both. So there is the edge and there's a cloud platform.

26:25
David Bader: So the question, just to repeat the question, are we providing a software stack just for the device or is there a cloud platform? And Dave's answer is yes, it's both, right? So there's a component that loads to the software or to the device itself, and then there's a cloud-based product that allows you that remote device... The remote access and the remote device connectivity. Yep. And that, you know, we can shape and manipulate that depending on the scale. I mean, the whole concept going back to that first slide is, is we wanna be able to do this at scale, right? If it's one piece, two pieces, that's great. We're all for helping with that. But if it's a 100 or 200, we wanna make it super easy. See, I told you there's always somebody that leaves early. I'm just kidding. Any other questions? Come on. There's gotta be. Yes, sir.

27:13
Audience Member 2: So your own network that's used to update the unit themselves, can they, you can you use your own wire in the private cloud network as an input?

27:25
David Woodard: Yes. Yes.

27:25
David Bader: So let's repeat the question. Can we load our remote access capability onto a private cloud or onto a server or something like that? Yes. You know, it's Dockerable, it's containerized, and you can load it almost anywhere. Yeah.

27:41
David Woodard: Yeah. We have use cases where the cloud's actually running just in the factory. Like just, no, none of the data leaves that factory. It's all isolated there. So we just run the cloud directly in that factory.

27:49
David Bader: Inherently, you know, out of the box, it's running in the cloud, it's running in AWS, but we can work with you to do it anywhere. Yeah. And in fact, we have customers that build other components that buy that capability and load it onto their devices themselves and run it in their private cloud so it doesn't have to just be on our device. Yes, sir.

28:15
Audience Member 3: This is probably a dumb question, but does this only apply to your end devices?

28:22
David Bader: No.

28:24
David Bader: No. So that's what I just said.

28:25
Audience Member 3: Some of yours, but I might already have an installed base of a thousand of some other manufacturers, Linux based, whatever controller, can your software widget be configured so that I can use your software to just secure everything?

28:42
David Woodard: It is my favorite answer in the world, and I'll say it depends. So if you want that level of security, it would require you recertifying those devices. So at least taking one and saying, "Okay, we put your tech software on here, we've done all these same steps, but you have to have it re-certified."

28:56
David Bader: That's a 62443 requirement.

28:57
David Woodard: That's not... That's just a requirement. But what I'll say is, as long as your box is running Linux preferably, but we have the ability to understand how your operating system works and that we can tie into it and make these changes, then yes, we can run on other people's hardware relatively easily.

29:14
David Bader: Yep. Not a dumb question at all quite frankly.

29:15
David Woodard: No, it's a great question.

29:16
David Bader: It's a really good question.

29:17
Audience Member 3: I might not want to replace everything.

29:19
David Woodard: No...

29:19
David Bader: Exactly.

29:19
David Woodard: Yes. And if you don't need that level of security, we can also run in Docker, right? So if you just wanna deploy it and use it for remote management and use it for some of the, some security features, but not all, that's a really easy way to deploy it and at least try it and test it and see if it works.

29:34
Audience Member 3: You sell your software...

29:36
David Woodard: So that would be a service you... We don't do, right? So that is not what we get into. So we actually, we use two external companies. I'm blanking. Two, Nord is one, but I can't remember the other. But we do our own audits, so we have to send products to them. It is periodic, I think it's three times a year, four times a year. We send them new devices, they test them to say, "Yes, you're still certified." So you would have to do something similar. And those companies, and that's how they make their money, right? So it's...

30:01
David Bader: Yeah, I thought it was in here somewhere, but it's not. So yeah, I mean that's a normal thing that customers do on a regular basis.

30:08
David Woodard: But we do have contacts with these companies, so it's also, we can help you like at least make those contacts and have that discussion.

30:15
David Bader: Yeah. There's one down here.

30:18
Audience Member 4: So that's the box that you... It's also running Ignition?

30:21
David Bader: No. No, go ahead. You go.

30:23
David Woodard: So this is more... This was the first device that we certified. But this is definitely more of a what you call like a gateway or like a, I say gateway in an Ignition audience. You all think about the software, but this is like a hardware gateway. So this would be running very close to like your PLCs and stuff. You know, the box we have that's running Ignition is a more server class and if you come to our booth you'll see it. So it's got like a Intel processor. It's got more resources. You can run Ignition on this but it'll probably more for like the Ignition Edge product. Yeah.

30:56
David Bader: Yeah. Okay. We have time for one more question. Alright. Looks like it.

31:02
Audience Member 3: Sorry to ask a second question. So I'm with the utility grid sector and you have IEC 62351 TC57. Have you ever heard of NERC CIP?

31:13
David Bader: NERC CIP? I have not. NERC, NERC, but not NERC CIP. Yeah. No, but I'll tell you what, if you would, if you take a minute and when we're done, we'll go back out there and I'll write that down and we'll get you some answers on that because it just may not be on this slide and I may not have run into it. Yep.

31:34
David Woodard: There's so many certifications and regulations that's... Especially in energy, it's...

31:37
David Bader: And that's why I put this slide into deck two is to tie it all together around that core. But I think we hit the time almost perfectly, right? So we're good. We're, again, the booth is right across from this door. If you guys have any other questions or if you want people to send you a lot of emails, come see us. Yeah. Thank you.

31:55
David Woodard: Thank you.

31:56
David Bader: Great job. Great job.

See firsthand how UK charity SERV Kent uses Ignition to create an AWS cloud-based volunteer management system to revolutionize its medical transport operations. Driven by Ignition Perspective, this application replaces archaic manual processes with intuitive interfaces featuring real-time geolocation data transfers, GDPR-compliant security, and optimized volunteer, vehicle, and product management. Hear Chris Taylor discuss the project’s challenges, solutions, progression,  and future enhancements and breakthroughs that will bolster SERV Kent's mission-critical endeavors.

Transcript: 

00:02
Bobby McKenzie: All right, welcome. Hello, my name is Bobby McKenzie. I'm the Senior Training Manager here at Inductive Automation. Maybe step away a little bit. So welcome to today's session. This is "How Ignition Saves Time, Money, and Lives for Medical Charity." I'll be your moderator today. To start things off, I'd like to introduce our speaker here. This is Chris Taylor. He's the Managing Director at BIJC Limited. He has extensive experience in automated control systems with the power industry. And since 2011, he's been a strong advocate of Ignition software. Personally, he's held senior roles in emergency power systems, critical power systems for global banking, energy and data centers. And he founded BIJC Limited in 2014 to develop Ignition SCADA solutions, and has since expanded into MES and edge-driven systems for various industries. Chris has been a member of the Institution of Power Engineers since 1997, and the Institution of Engineering and Technology since 2005. All-around great guy.

01:10
Chris Taylor: Thank you very much. Good afternoon, everybody. As Bobby said, my name's Chris Taylor, and this is just a talk about BIJC and a charity that we support. That's me. So the charity that we support is called SERV Kent. It's an emergency courier charity based in the southeast of the UK. What we do basically is we are couriers and we transfer blood, blood products, milk, that's human milk, medication, samples, anything really that will go in a car or on a motorcycle. And we transport it between hospitals and hospices and blood banks, patients' homes, anywhere, really. And the reason we do that is because out of hours the only alternative is an ambulance or a police car, something like that, an emergency vehicle. And that takes a very vital resource out of action.

02:23
Chris Taylor: We also supply blood to our local air ambulances, either at their base or on scene. And typically that's what I do. The services are free and it saves hospitals funds for patient care. At the moment, we have 200 volunteers. They cover their own expenses, and we have typically about 6,000 calls a year. SERV Kent is recognized by major hospitals as an essential service for saving lives. Personally, I hold advanced motorcycle and roadcraft qualifications, and for me, volunteering for the charity is a great excuse to go for a ride on my bike.

03:12
Chris Taylor: Okay, so why SERV and BIJC? So we're a Premier Integrator, and when Ignition 8.0 was released, 8.0.0, we really needed a a project to learn Perspective on. Now Inductive University is great, and all of our new starters go through the university, but we're a great believer in having real-life challenges to learn from. And so we decided that as a business, we would support SERV Kent by creating a project or a set of projects that they could use in their day-to-day operations. So they had a few challenges. Before we had the Ignition projects everything was paper based. Everything in the charity was paper based, but it was in all different locations. So that was a challenge. Product deliveries were done with a little booklet with three different colors, bits of paper, a pink, a blue, and a white. And when you collected, you have to give a white one out, when you delivered, you have to give a blue one out, and there were 25 of those in each book, which is fine except for some volunteers might only do 25 runs in six months.

04:39
Chris Taylor: And so we would only get that information in six months. And so our data collection was really, really poor. The slips were illegible, they'd get lost, and so we'd have huge gaps in the data. So that was our really, that was one of the first things that we wanted to sort out. Secondly, was a rota system. With 200 volunteers, you need to know who's available when, where, and what they can do. And so we wanted to build a self-service rota and also for volunteers to be able to see when they could volunteer. Volunteer management with 200 volunteers, we need to know their addresses, their phone numbers, what their qualifications are, all of the things that you need to manage. This was already done, but it was a filing cabinet. It was in one location, and if more than one of the committee wanted to access it, there were security issues, GDPR issues, all sorts of things that we needed to get right.

05:47
Chris Taylor: Fleet management, we have a number of cars, and we have a number of motorcycles that are owned by the charity. So we need to know where they are, who's got them, who's allowed to ride them or drive them. Do they need servicing? Do they need insuring? Do they need tires? What's their mileage? All of those things. And finally, event management. We needed a way to let our members know when we are having events for fundraising, when we were having events for awareness, if there were social events, we do do some social events. So you can understand that there was... These are just five of the basic things. There's an awful lot more, but you can understand that, if we could have a single system that allowed us to do this it would be fantastic. So this is what we decided to build. And we started in, the first bit went live in 2019, in September 2019, so pretty much five years ago.

06:53
Chris Taylor: So the first thing we built was the rota system. So this allowed members just to click on where they want to volunteer. Again, with most things, it looks very simple, but in the background, extremely difficult because different members have different qualifications. They operate in different areas. Some members are controllers, some members are riders, some members are drivers, some members are both. Some members can do the air ambulance, lots of things. So for Perspective, we were very used to building in Vision. So this was our first Perspective project, and we tried to transfer our Vision skills to Perspective, and we very quickly learned that wasn't gonna work. And so we had to learn a bit of web dev stuff. And so for example, this rota, when we first built it, we built labels and we put icons with the labels, and then that was a view.

08:03
Chris Taylor: So if you look at this page, for example, we've probably got 20 views on this page, just on that two rota slots. Sounds fine. But in the initial versions of Perspective, we found that the icon library was per view, so each, sorry, per, yeah, per view. So when you put an icon in a view, it downloaded the entire library. Well, if we put an entire month on the screen, we could be downloading the icon library 600 times, and it kind of slowed things down. So first of all, we learned how to build really small icon libraries, bespoke icon libraries, but Inductive Automation as they are, saw that this was a problem. And I think by 8.1, it was an icon per or icon library per page. So they've sorted that out. But it was a pain point and it's, when you start with a new system, you have to learn these things. But now, so this is what it looks like these days. It looks exactly the same to the members, but the way it works is completely different.

09:25
Chris Taylor: For example we learned about binding refresh. And that's a game changer for this because initially when someone clicked on one of the icons to put their name in, it would refresh the entire data set. But now, well, within a few weeks of us deploying this, we worked out that you could use the binding refresh and just change the users or the label that we were dealing with. And it was a game changer because the user experience, and you've got to remember that the users that I'm talking about, they're not in a factory. They are just normal members of the public who volunteer. They don't understand what's going on in the background. And so their user experience is really important. And this project really helped us hone user experience.

10:27
Chris Taylor: So the next thing that we really needed to work on was collection, delivery, and transfer. Again, before I spoke about the booklets. And so we developed a smaller project that was more mobile friendly. And what we wanted to do was to be able to collect the data. And one of the important things is we needed to know who, when, where, and what. They were the four things that we needed to capture. And with location services, Ignition allowed us to do that really, really simply, so we could get the location, when they hit the collection button, we asked them, what are you collecting? And that could be multiple products. And then they would just set their destination and hit the collect button, and that would create an entry into the database. And then when they delivered, they just, if they were in the right location, it would say, "Oh, it looks like you're there." Push the button. And it was delivered.

11:31
Chris Taylor: And what that meant was, whereas before we were collecting data every six months, if at all, we were collecting data in real time. And it was a huge difference to our stakeholders because they were able to say, "Okay, where's my product?" And we allowed our customers or our stakeholders at the hospitals to connect into the system and actually see where things were. And, huge, huge difference. One of the things I want to talk about with this though, is transfer. We found out in the first year that we were losing a little bit of data when we transferred. So to stop volunteers from having to travel too far from home, the area that we cover is the whole of the southeast of England. You can easily be 200 miles away from home at times. And so to stop people from traveling too far from home, we operate a relay system. So they collect from hospital A and they'd go to a meeting point, swap it over, go to hospital B, which is fine. But in the first iteration of this, they had to end the delivery and then start a new delivery. So we wanted a method to transfer from one phone or one mobile to the next mobile. And what that meant was that we had to figure out a way of opening or opening a pop-up on one session from a different session.

13:07
Chris Taylor: And it's a powerful thing once you get the hang of it. But again, not straightforward. So these are a few tables. This is probably about 20% of the tables that were involved in a delivery transfer. But, again, it's never as simple as you think it is, because when we collect products, we are not generally collecting one product. We might be collecting three products, and when we transfer them, the destinations, they're all collected from hospital A, but then they're transferred and then they've gotta go to hospital B, C, and D. So we have to be able to track all of that in the transfer. So the process for this is, the first thing that we have to do is we have to find out what is the session that... What sessions are available, who is logged on to a Perspective session. So we use system tool, get session info, and that gives us all of the sessions. We can then filter that for the project that we're interested in. And then from that we can use the location data for each session and find out how many people or what people are within about 500 meters of the initiating session.

14:31
Chris Taylor: From that, we can get the ID of each user, and from that, we can get there. We can get a photograph of them, because with 200 volunteers, they don't know everybody. And so it's much easier if you've got a photograph so they can see who they wanna transfer to 'cause they're usually standing right in front of them. And they poke them in the face and that initiates a session-to-session message. We don't actually need a big payload in that message. We just need the run log ID number. And then the recipient receives a pop-up. They just click, I'm gonna accept that. And the transfer is complete. What we do is, it's just a... So this is, sorry, I'm going ahead of myself. So they get a transfer request, they click on that. And from there we, sorry... We don't transfer much data, but what it does, it initiates a series of name queries to end one run, start a new run. But importantly, link the two runs together, both from the recipient and from the initiator. And in the four years that that's been running, we've done 4,500 transfers, and we have lost zero bits of data. So we are very, very pleased with that. I do have, I don't have. I was gonna show you a video of the transfer happening.

16:19
Chris Taylor: It's, I think, it's been made into a still. But anyway, so what the video shows was, you can see there is a "transfer to" button. You press that button and you poke the person, and it really, it transfers in three, four seconds. And no loss of data. If anyone's really interested, I can show you on my phone. So the other thing that we developed was a calendar. And you can imagine running a charity that delivers things. We needed to schedule quite a few things. And there is no calendar option in Perspective. So we built our own. I'm very fortunate in the business that I have people who can build modules for us. And we wanted a calendar that was easy to use. And so we implemented a copy of a FullCalendar.io as a component. And it has everything that you would expect. Outlook, or Google Calendar to have. You can click on it, you can create events, you can create reoccurring events, all of the things that you would expect a calendar to have. We use it, so for each member, they think they have their own calendar. It's just a data set. Each time they open their calendar, we search for everything that's relevant to that particular member and display it on their calendar. If I go back, you can see, that has like fundraising and a talk to people. There'll be social events there, all sorts of things. It's very, very useful for planning.

18:25
Chris Taylor: We've had a look at that. Oh yes, we can export any events as an iCal object, which is very useful. The other thing that we really, really need to do is mileage tracking. It's something that we were never able to do before. And for our stakeholders, it's vitally important for them to know what we are doing. And so they can calculate their savings. We have around about 50 named locations dotted around the southeast with four major trauma centers. And again it's quite complicated for us to calculate the mileage and the time it takes because each member is started from a different location. They're generally starting from home. And then they're going to hospital A, hospital B, hospital C, and then back home. That's a simple run. We can gather the data from the run log that they fill in. But of course, if you just look at that data, it looks like they're traveling from hospital A to hospital B, hospital A to hospital C, hospital A to hospital D, and that's not the journey they're taking they're going A, B, C, D. So we've got some rather complex scripts that go through that look at each, each entry in chronological order, see if they're close to each other in time.

19:58
Chris Taylor: So when they were started and then tries to work out a route. And it works about 95% of the time, we get it right. It's extremely tricky. Once we've done it, we stick it through the Google API. So we have to, each of the run components, we have to get the longitude and latitude. We stick that into the API, and that gives us a distance and a time. And then we have to build the run up again that way. We end up with a huge dictionary of longitudes and latitudes and times. And that's a script that runs once a day for every journey. But as I said, we don't always get it right. So we have a manual check that members can use to work out their locations. They can enter everything in manually and just check. And if we're wrong, they let us know, but it also gives us information to just refine our scripting. That's a little bit of scripting, which doesn't mean much, but it's something to stick up there. I could talk for a very long time, but I was trying to find things that were of interest, because a lot of what Ignition is doing for SERV Kent is just quite mundane. A database entry or a front end of a database, but it makes such a difference.

21:33
Chris Taylor: A couple of photographs. This is, that's Mr. Shepherd there with his bike and my bike. This was about a month ago, quite a rural area. We were called here because some pedestrians thought it'd be a good idea to cross that road. That's a 70-mile-an-hour road and they both got hit by a lorry. Two air ambulances in attendance. This is just as the second ambulance had taken off. And Gary brought blood from one hospital, I brought blood from another hospital. So that's the kind of thing that we do. So this is not, we did quite a lot of mundane things but when you've been with the charity a long time and you've got a bit of a few qualifications, that's the sort of thing that we do. And Ignition makes this work because we know who's available, when they're available, and where they are. So this was in the early evening, so we were both out of work, but the controller knew we were both free, knew where the vehicles were, and knew where the closest hospitals for us were, and were able to contact the hospitals, get the blood prepared, and we were on scene within, I don't know, 35 minutes of being called. So pretty good. There's our shiny motorbikes, they were clean there. If anyone's interested, they're a pair of Yamaha FJRs, police spec motorcycles with emergency equipment and lights and sirens, that sort of thing. And go-faster stripes, we like those.

23:26
Chris Taylor: Just a few bits of information here. So this is another thing that Ignition does for us. We have an infographic. I just put a start date and an end date in it, and it calculates this for me, which takes it about three seconds to do that. So since we've been using Ignition, we've done over 30,000 runs. We've traveled over two million miles, and we've volunteered for 90,000 hours. There's two things I wanna say as well. We benefit from the nonprofit credits for, that AWS give us, for a small fee, we get $1,000 of credits, which just about covers our hosting in AWS. But more importantly, Inductive Automation last year started the Community Impact Program. I think we were one of the first people to sign up for that. And now I don't have to worry about the cost of upgrades. Fantastic. So that's been a huge help for us. That's it. Anybody got any questions?

24:38
Audience Member 1: You were the first. You were the first.

24:39
Chris Taylor: Was I the first?

24:43
Audience Member 2: Howdy. First off, it's really neat for you to do something impactful or something that we all do largely for customers that make millions and billions of dollars 'cause you're actually making people's lives better or longer. Secondly, how did you acquire the data for the the people and the equipment, like, what means did you get the GPS location?

25:07
Chris Taylor: So GPS location comes from their own devices. So they use their own cell phones, mobile phones, and Ignition even just as a web browser, 'cause we don't typically use the app because it's a little bit complicated for members of the public to use. So it's just a web page. But from that web page, we can still pull. We just tell them to turn location services on, and we can pull that through. Do have a bit of a problem with Androids now and again, but we can get that information through. As far as the product that they're picking up, when they select a collection, it brings a pop-up with a whole list of things that it possibly could be. And they can select that.

25:57
Chris Taylor: In the near future, we're gonna have to do barcode reading for blood so that it's got more traceability. And I'm really looking forward to 8.3 for offline integration because that is a challenge that we have if a member is somewhere where there's no cell phone coverage they have to do it manually when they've got coverage. Because, although I've been describing collection and delivery is fairly automatic, they forget. So if they go to deliver something, they go, "Oh, I forgot to collect it on the app." They can enter it in manually. They can go to a manual mode and enter it in. Yeah. So for GPS location, they just enable GPS services and for... They can select what they're collecting. But importantly, we only track people when they're using the app. That was something we had to make very clear to them because we did have a bit of pushback 'cause they thought, "Oh, you're gonna know if I'm speeding, you're gonna know what route I've taken." So we only, basically, it's only when they're interacting with the app that we know where they are.

27:14
Bobby McKenzie: I actually had a question for you. So you mentioned at the very... Whoa. Let me stand over here.

27:22
Chris Taylor: Probably.

27:23
Bobby McKenzie: So you mentioned that you use AWS, at the end there. How do you use AWS with this app?

27:28
Chris Taylor: It's very simple really. We used to have a, we've got an instance in EC2, just a very small, I think it's a T3 now, we had to upgrade. T3, very small instance and we have a MariaDB instance as well. We just connect the two together. AWS makes it really, really easy to do. I think the most tricky thing is the certificates, getting the security certificates in. But once you work that out, it's very straightforward. I mean AWS do make it really simple to do simple installs like this. It's very straightforward. And we did this in 2019 when things weren't quite so easy, but the most tricky thing was setting security. Oh, and the DNS, the DNS wasn't quite so straightforward, but this has a separate URL just for this. But our members don't really see that because the just log on and they can log into the main site, hit run, and it takes them directly to the other site. So yes, but AWS has been great. Our uptime is phenomenal.

28:42
Chris Taylor: I mean, I can't remember... One time I did an update to Ignition and it failed. That caused me a little bit of grief. But that was a connectivity issue, that wasn't Ignition's fault. So that was the only time we were down for about 30 minutes. Other than that, I don't recall any downtime. Very good. Yeah, and we have to have it running all the time, 'cause we run a 24-hour operation. And we don't have many users on there. I think the most I've seen is about 35 people on it. But that's because people are volunteering in rotas. Yes, sir.

29:27
Audience Member 3: Have you shared to other parts of England?

29:34
Chris Taylor: So yes, they know about it and they've seen it. The thing is there, so there is a group called NABB, which is the National Association of Blood Bikes, and there are other groups, but we all operate differently. There's no single... We don't operate in the same way. We have different rules. And then there's GDPR. Everybody would have to be a member of everybody else's group for us to share information. We did look at rolling this out as a project that they could use independently, but they don't have the skill set to manage it. And I don't really want to be managing 20 different versions of this. We did look at going nationally, but again, they operate differently, so it's difficult to do. One day, perhaps.

30:47
Bobby McKenzie: Any other questions? Well, I do have one very important question for you. Are you guys gonna win the Build-a-Thon?

30:55
Chris Taylor: Are we gonna win the Build-a-Thon? So I have three of my, I say guys, three of, two guys and a girl who are currently at headquarters working very hard. Are we gonna win it? I don't know if we'll win it, but we'll have the most fun.

31:14
Bobby McKenzie: There we go, love it.

31:16
Chris Taylor: Yeah. We've made sure that we're gonna enjoy it and there's a few jokes there, so.

31:21
Bobby McKenzie: Excellent. All right. If there are no other questions, then we can end there. Thank you very much, Chris.

31:28
Chris Taylor: Thank you, you're welcome.

Related Content

• IA’s Community Impact Program: Helping Nonprofits Help Others

In this session, 4IR Solutions will showcase best practices and technologies to rapidly deploy and remotely manage large-scale Ignition systems in the cloud and on-prem across hundreds of sites. We'll demonstrate zero-touch provisioning and real-time updates to a fleet of Ignition installations.

Transcript:

00:00
Kyle Van Eenennaam: He also has been a user developer and integrator of advanced manufacturing software. Randy brings with him a deep understanding of intelligent edge solutions and a proven track record of global on-premise and hybrid deployments. Over on my left, I'm fortunate to introduce James Burnand, the CEO of 4IR solutions, who's a seasoned veteran of the industrial automation industry, boasting more than two decades of experience as well, leveraging his experience now to empower manufacturers by providing the infrastructure for their plant for applications to thrive in the cloud is a keen understanding of the intersection of cybersecurity, operational requirements and management, which we might boil down to Industry 4.0 and James Craft's tailored solutions for companies embarking on their cloud enabled and highly automated OT journeys. So without further ado, please welcome both of our presenters to the stage.

00:52
James Burnand: I think I'm mic'd up, right? So thank you for that introduction. I'm not sure whoever wrote those profiles, but we need to send them a nice gift basket. So the agenda for today is we're gonna talk a little bit about enterprise architecture. So we've got some pretty exciting demonstrations near to the end of the presentation. But in the beginning, we wanna talk and describe a little bit about what we mean by enterprise and what distinguishes that from a non-enterprise or more of a standard sort of a solution. We're gonna go through a bit of the design principles and the considerations that you need to make when deploying in an enterprise level environment and really talk through how that can stretch and be extended down to the edge. So have you ever needed to deliver across multiple sites or started with a small application that all of a sudden became a big application, and once people saw what it could do, there was this huge demand to deliver it consistently across a bunch of different locations or a bunch of different users, or been needed to take something from ignition and tie it in with your enterprise systems.

01:56
James Burnand: So most commonly it's ERP or other enterprise level systems, and all of a sudden it becomes more complicated because now we have to meet not just the regulatory and operational requirements of operating inside of a facility, but also of in operating inside of the IT environment and potentially within the cloud, depending on what the solution is. So if, so, we think you might need an enterprise architecture, and yes, I did that animation myself. So what is an enterprise architecture? So really it has a handful of characteristics, but it's not something you can really just define as specifically this, or specifically that. You really have to look at it as a set of characteristics that make something enterprise. So usually it's got a greater need for uptime and security, and that it's a more complicated, sort of a deployment that often will span a wider range of people, locations, or application integrations.

02:51
James Burnand: Usually these systems are involving a variety of different departments and a variety of different individuals and auditable requirements to be able to be deployed. So they tend to be a lot longer to identify and a lot longer to deploy, but they tend to have a useful life that can deliver tremendous value for an end organization. So what do we see as some pretty common enterprise applications today? So, unified namespace has become a huge endeavor for a lot of organizations really around digital transformation and being able to apply these new digital technologies to the way that they operate inside of their facilities. And oftentimes that will represent itself as big data systems, AI systems or systems that are deployed to be consistent across a variety of different geographies, locations, facilities, lines, whatever, whatever the characteristic of lots of you're looking at, the enterprise applications are often delivered in that way.

03:51
James Burnand: So it's important when we talk about enterprise systems to distinguish what we think are some of the best practices for deploying enterprise systems. So we've categorized those into to five different groups around security, reliability, scalability, operational excellence, and cost optimization. And I'm gonna describe really from a cloud centric point of view, what does that mean? And then Randy, when he takes over in a couple of minutes here, is gonna talk about how that extends down towards the edge. So for security, some of the key design principles in security are things like identity providing, so authentication authorization, so the use of identity providers and individual logins for your users using things like multi-factor authentication, role-based access control for easier management of lots and lots of different users that need to have specific capabilities. So different than setting up the admin user for all the different gateways is setting up admin groups and then being able to grant and revoke the access as is necessary for those users to perform the tasks that they need to on those different deployed assets.

04:57
James Burnand: Certificates. So specifically when you're talking about the cloud, certificates become a must. In the way we do our deployments, we actually use public certificates, which implies that those certificates are known and trusted by all computers. That requires a little bit of work and a little bit of automation to make it work well. But that also means that we have the ability to rotate those certificates on a regular basis. So what we see in a lot of end users is they'll set a certificate and however long they're allowed to set it for is how long it gets set for. What we do is we actually rotate them every 30 days, and that's a security best practice. From a security principle perspective, that's important when you're exposing something to the public internet, which oftentimes cloud applications are. It's important to encrypt it in transit and at rest, which obviously SSL is a big piece of that.

05:45
James Burnand: But also when you're doing things like setting up your MQTT network or if you're setting up your communications with other systems everywhere you're able to encrypt, you need to encrypt as part of these applications to minimize the surface area for potential attack. And then finally, audits. Everyone loves that word, but it becomes absolutely critical in these enterprise systems, especially when they span across a wide group of people in a wide group of users as well as administration folks, that there's some level of auditing going on on those systems to ensure that the security posture that you have created when you first deploy it is maintained and improved over time and that, "Hey, I just went in and added this admin user to get me access for this piece." Those are the kinds of things that will get caught in a periodic audit and make sure that those things are removed. Should they have, should they somehow escape the task list of the person that added them.

06:45
James Burnand: So next we wanna talk a little bit about of reliability. So reliability and availability are somewhat similar. Availability really comes down to the choice that you make for the application and the user experience that you're trying to create. So for reliability and availability, you really need to think about what is this application and how much of this application, how much downtime can this application tolerate? The nice part about the cloud is that you can make it extremely available. You can also empty your pocketbooks by making things extremely available. So you have to figure out what the right balance and tolerate, and ability to tolerate is for that specific application, and then ensure that the way you've configured the assets and the way that you've configured the services that you use is appropriate for that use. So for everyone's benefit, one of those pieces is if you're deploying in a data center with multiple availability zones, what an availability zone is, is it's a building, it's got its own power supply, its own internet connection.

07:46
James Burnand: And its own set of services is a part of it. So it's essentially as good as a single data center, in most of the regions or most of the bigger regions for the main cloud providers, their regions actually have three availability zones in them. So when you deploy certain applications at certain availability, they're not just mirrored on multiple hard drive arrays and multiple hosts like you would get in a VMware system, they're actually mirrored across multiple buildings. You would literally need to take an entire building out to see a minute of downtime. So understanding what that means and how that affects the pricing, but also what that availability target is, is an incredibly important exercise as a part of architecting and building out an enterprise architecture. And then finally is network. Network is a little more complex than just your internet connection, but understanding what level of reliability you have in communication from a site into a cloud-based environment so that you can make intelligent choices about what it is you need from an availability perspective.

08:48
James Burnand: So one of the great technologies that Inductive has as a part of their platform is Store-and-Forward. So Store-and-Forward is a key component to handling poor internet connectivity. Some applications can't tolerate that. Some applications need more more persistent connectivity, and that's where you get into more complicated SD-WAN and redundant connections to the cloud. And certain organizations at an enterprise level have already taken that journey and others haven't. And really understanding what that is as a function of what the needs of the end users are gonna be is an incredibly important part of the architecting exercise for an enterprise application. And then the final thing is, is actually something we have learned, I would say over the last couple of years is, test environments for cloud-based resources.

09:41
James Burnand: So different than DevOps around Ignition, where you have a development environment and a test environment and a production environment. One of the things, one of the characteristics of a lot of enterprises is they have a lot of security policies and changes that they make at a tenant or a subscription level inside of their Azure and AWS environments. Those changes can have very major impacts on the workloads that are running inside of those spaces. So it's a really smart idea when you're building out an enterprise application to also consider building a duplicate of it so that those policy changes and those upgrades to the services and things that are happening at a cloud provider level can be managed and tested before they're applied to a live working environment.

10:27
James Burnand: All right, scalability, this is my favorite one because scalability is a little less, oh, sorry, I didn't actually advance the slide. Lemme turn that thing back on. So scalability is really more about making or avoiding decisions that are gonna limit you. So what we find in scalability is, is it actually starts with the application. So oftentimes when you start building an application and Ignition, you get a gateway and you start connecting tags and you start building perspective displays and you start connecting to databases, then all of a sudden you've built something and it's great, and then all of a sudden you need to add 100 more users and five more sites and all of these pieces to it. And all of a sudden it's like, "Well, that's not gonna fit on a single gateway. I really need to think about now how am I gonna do this? Or do I need to have dedicated gateways for each part of this application and how should I do this?"

11:18
James Burnand: So we actually encourage people when they're starting off, especially building small applications, to think about things like the scale out architecture as a potential eventuality depending on what the use case for this application could be. So that's things like separating your front end and backend into separate projects in the same gateway so that if you ever do decide to pull them apart, it becomes a much easier exercise in the future. We also recommend the use of EAM and inheritable objects and components. So being able to take things like your navigation bar and certain project components and tags and be able to have those as common pieces that you can then pull down into your environments allows for you to manage those pieces centrally, but deploy them wide across all the different use points for them.

12:05
James Burnand: And finally, around scalability, we want to talk about the cloud. So there are services in the cloud that make scalability very easily. So things like databases, load balancers, storage, compute and orchestration. These things allow for you to be able to start small and add as you need the capacity. Now, being able to do that also requires that obviously your application architecture makes sense, but that you make intelligent choices about what your potential future state could be. One of the key technologies that we at 4IR have adopted and we think is crucial to enterprise architectures in general is containerization. So for those of you that haven't done Docker containers and played with the stuff that Kevin Collins has published, please do. It's, makes your life so much easier when you're testing and building if you have, this conversation will hopefully make a lot of sense to you.

13:00
James Burnand: But being able to create a very easy unitized way of deploying Ignition and the different components that are required and configuring it is a really scalable way of doing the operations. And what you'll see on the screen there is I have a cow and a cat, and this is intended to represent what they call cattle versus pets. And so if you think about a virtual machine that's running Windows as an example, we consider that to be a pet. So that means that you have to manage it with a level of attention and care because it requires it because you have to perform individual updates and you have to manage it more directly. Whereas containers are really more like cattle where we treat them like cattle, and if something goes wrong with one, we're not taking it to the vet and paying a huge bill to get it's hip replaced. We're getting another cow.

13:56
James Burnand: So there's much better analogies online that you can read about that. But that is, in all intents and purposes, how we treat the containers that run ignition is we treat them as cattle, as a herd of capabilities. And I think when you see the demo that Randy's gonna be running today, hopefully you'll see what that looks like in action. Alright, Operational excellence. So really this comes down to that you can deploy all this stuff, but if no one's watching it and updating it and looking at it from a compliance perspective, something will go wrong. So as part of your planning for when you're going to rule this out is you can't forget that the continuous improvement and updates and patching and monitoring and alerting and having the right support mechanisms in place are absolutely crucial to the experience of the end users. And finally, everyone's favorite cost. So cost optimization is something, obviously it's related to things like availability, but using technologies that make sense for you to be able to manage your cost in the best possible way. So as an example, when we do backups, we do backups in the cloud because it's really cheap to store stuff for a long time and cheap storage in the cloud.

15:09
James Burnand: But we also do things like we consolidate services. So I don't need an active directory system in every location if I can have a central system. And then basically the ability to operate and use that from a bunch of locations. Same thing with databases, same thing with a bunch of different technology pieces that allow for you to make it easier to provide these services that you need as a part of the application deployment at the best possible cost ratio. And finally discounts. So understanding how discounts work in the cloud providers is tremendously important. So it's a lot cheaper to buy a virtual machine or a cluster if you say, "I'm gonna use this for the next year." You get a pretty big discount for doing that. There's also negotiation, there's spot instances and there's pay as you go for certain other services that allows for you to really optimize how much your cost basis is for the different services that you need.

16:07
James Burnand: So to talk about edges and clouds, our friend William Shakespeare, it's a cloud or not to cloud that is the question. It may seem like we would say the cloud is the answer for everything 'cause it's like literally in our logo and our website name. But we recognize and we don't encourage every single application to be deployed in the cloud. There are certain considerations that you need to have around things like the user experience tolerance to internet connectivity, latency, and whether or not the staff skillset is able to handle some of the different pieces and parts. So we recognize that there's a very strong use case for being able to and to have things that operate on-prem. We also happen to have an idea and a concept that makes that a little easier to do at scale.

16:56
James Burnand: It's called hybrid cloud. So hybrid cloud is like taking a bit of cloud, paving it off and putting it on a server in your building. There are a variety of different providers out there that do this. I would say this is, from my perspective, I think this is gonna be a huge growth area inside of the manufacturing space in the next several years. Not a whole lot of people are doing it yet. But what it gives you the benefit of is that you can now operate and manage your on-prem resources using the cloud portal and the automation and the tools that are available to you as a part of your deployments in Azure or AWS or Google Cloud. So it kind of gives you the best of both worlds. But like I said, it's still a relatively new technology, but allows for you to have that capability of central management and local operation. All right, with that, I'm gonna hand it over to my partner Randy.

17:54
Randy Rausch: Thank you. So James just took us through the on-prem, or the design principles for in the cloud. And we're gonna talk about, they are for on-prem. Surprisingly, they're the same thing. We're gonna do it again because they're a little different when you get to the edge, what do I mean by edge? If you're a person from the cloud, the edge is where the data originates. It's your facility, it's your site. So there are a few things different when you start working there. In particular security. So what's different, the physical security and your network posture is different. There are some data centers that will have armed guards standing outside, making sure physical security is pretty good. That's not always the case for maybe a server or a IPC that you have on site. So you need to deal with things like that. What can you do?

18:36
Randy Rausch: If you are remotely managing something and you don't have physical control over it and other people do well, you wanna make sure someone can't come in and plug a USB stick or start logging into your terminal. So go ahead and disable interactive login because you don't need it and it's just yet another threat vector. Another thing you can do is utilize the security of the built-in hardware security modules or your TPM trusted platform module chips. On top of that, you can do secure boot, you can use that to encrypt your hard drives and make sure things are encrypted at rest and in transit of course.

19:10
Randy Rausch: And you wanna minimize your attack surface. So in this case, for your operating system, if you wanna rip out everything you don't need, because that's an attack factor that, again, you don't need. In our case, we use network, 'cause it has to talk on the network, and we run containers. So we have networking containers in the operating system, and that's it. Significantly reduces the attack surface. From a network perspective, you've got a different security posture in that the worst thing you can do is be a vector for some bad actor to come into your customer's network. So some things you wanna do from a best practice is you should never have any open firewall ports. All communication should be outbound from your on-premise device, typically over port 443 outbound only. You wanna make sure you're using your chain of trust. So again, your TPM chip on the hardware, all the way up through your certificate authority in the cloud. You wanna make sure that your whitelisting or allowlisting the destinations that your computer is allowed to go to.

20:12
Randy Rausch: From a scalability and reliability perspective on-prem, there's not as many computers on-premises as there are in the cloud. And so from a scalability perspective, that can limit you. But on the plus side, workloads in your factory are pretty predictable. It's not as if you're gonna have 100 new manufacturing lines on Black Friday just for that day. So it's fairly critical. So you don't need that elasticity, but you still wanna architect for scalability as you will grow over time or at least give yourself the ability to. The fun part is when you start talking about reliability. So reliability and cost have an interesting relationship. Things fail, right? It happens. The question is how long can you tolerate a failure before you have to restore service? The answer to that is usually between 10 milliseconds and 10 weeks, right? So if you're using a server industrial PC that has a 10 week lead time and you don't have a spare on-site, you are gonna wait 10 weeks if something goes wrong. If you do have a spare on-site and you cannot tolerate how long it takes to swap over from the broken machine to the spare, then you wanna start to look at a high availability architecture.

21:30
Randy Rausch: In this case, it may be a hot spare running live. Maybe you need even more. So maybe you have a multi-node Kubernetes cluster to give you some of that availability. But it's important, a thing to consider. And it's not just the compute. You wanna look at it holistically. If you only have one WAN connection, that's a single point of failure. If you only have one power source, that's a single point of failure. You do wanna take advantage of or consider that entire design, right? So when you're designing, the primary input is your availability target, right? And that's, again, how much downtime you can tolerate. A 99% availability translates to seven hours a month of downtime. 99.9, it's about 43 minutes a month of downtime. Four nines, 99.99% availability, it's about four minutes of downtime. And if your customer says, "I need 100% availability." If you just follow the math on that, you can deliver that for infinite cost, so. All right, in terms of operational excellence, some things you wanna make sure you consider, updating regularly is so critical.

22:39
Randy Rausch: Easy to do in the cloud. You also have to do it on-prem or on the edge. This is for your security updates, but also you should be having new functionality updates and making sure you're regularly keeping things patched, up-to-date, adding new features, and you have that muscle built to regularly update. It's important to have full bare metal update capability. If you're managing this device remotely, the operating system needs to be fully under your control. It should be a single artifact operating system without package managing of that, that can cause drift. And you should have A-B updates, so you can entirely update the operating system, fail over to the other partition. If something doesn't work, you can go back to your first partition and restore service. You don't wanna brick your device, especially if you're not there to go fix it. Certainly you wanna have backups.

23:27
Randy Rausch: When you're on-prem, you wanna back that up to the cloud or some other offsite location. And you wanna have 24/7 monitoring, as well as the ability to do self-healing and spares management and other types of things. So, enough talking. Let's take a look at some of this working. So we're gonna do a demonstration. So here's what we're gonna see. We are going to forcibly kill an ignition gateway and watch the self-healing capability, bring it back to life and restore service. We are going to take and deploy an additional 23 gateways, commission them real live. So if that takes, if it takes you half a day to do that, this will be about 10 days until this talk is over. We are going to upgrade a set of our ignition gateways, and we are going to then downgrade to simulate as if we needed to roll back.

24:19
James Burnand: And just for everyone's clarity, what we're doing here is this is actually our platform that we operate called FactoryStack. It's using a technology called Kubernetes. So Kubernetes is an orchestration engine that allows for us to operate and manage at scale a variety of different containers. And then what Randy will describe is a little more detail behind some of the tools we use to automate those deployments and management.

24:44
Randy Rausch: Okay, a little bit about the architecture. In this case, we have an enterprise architecture deployed. We have several front-end nodes. We're starting with three, that'll scale to five. We're starting with several back-end nodes, I think two, and we're gonna scale that to three. And we're starting with several on-site, in-factory nodes, starting with 30, and we're gonna scale that to 50. All of these will be communicating through an enterprise broker, in this case, HiveMQ. And yeah, so it's a live demo, what could go wrong, right? All right, let's see. A little bit of orientation here. This is a live perspective application. Each one of these boxes represents a gateway. So if there is a gateway, it is publishing to a broker. If this application sees that topic in the broker, it will make an image for that gateway. If that gateway goes offline, it will turn red, right? Or if it stops communicating with the brokers. We're looking at the broker and what's happening, it'll turn red. So if new things come in line, they'll pop up. If they go offline, they'll turn red. Live application out in the real world with certificates and things like that. So yeah, let's do this live. Okay, first things first, let's demonstrate self-healing. So audience participation.

25:58
James Burnand: Which city do you hate the most?

26:00
Randy Rausch: Which city would you like to take offline? We can do more, we can do more. I heard of Paris, is that? All right, all right. Houston. All right, where's Paris? Okay, so if I come here to Paris, this is our gateway. You can see, right here we've got our live certificates. This is available, a public-facing endpoint. It's live. We're up here. What I have on the right-hand side, this is a system inspiration tool. It's not meant to be pretty. It's called K9s. It allows us to administer Kubernetes clusters. In this case, we're looking at the pods in our cluster. What I'm gonna use is this tool to forcibly kill the ignition that is running here called Paris. So here we go. Delete, are we ready? All right, so I have just completely removed the resources and the ignition gateway. If I come back here and I try to reload, gateway's just gone, right?

27:02
Randy Rausch: All I've done is kill it. The infrastructure that we have in place with its self-healing capability, it has recognized that something that is supposed to be running is not running, and it's going and auto-restarting and restoring the backups in the background without anything on our behalf. So if we refresh this again, we should see the gateway is starting up. And if we come back here, oh, I was too fast to see it red, but you see Paris is now back online. So a little bit of self-healing goes a long way. So if something does go wrong, you wanna recover quickly, and that's a pretty quick way to do it. All right, let's scale out. I am gonna start this, 'cause it takes a little bit of time. All right, and then I'm gonna explain what's going on. So in this case, we're using a tool called Pulumi. This is like Ansible or Terraform. It's an Infrastructure as code automation tool. We have given it a input of what we want the state of our infrastructure to be.

28:04
Randy Rausch: It then goes out and looks at what is actually happening in reality. What is the real state in the real world of what's there and compares the difference in what you want and what it actually is. Based on that difference, it will execute the code to make the real world look like what you want it to look like. So, command line tools may not be the most interesting things to look at, but let's see. You can, in this case, we are creating new things. On the right-hand side, again, this is the set of pods that were there. As the infrastructure is determined what needs to be there and we're refreshing what's out there, you're gonna start to see new pods come online. So.

28:45
James Burnand: So this isn't just like turn on ignition, right? This is going out and enabling and deploying the cloud services and setting up the drive volumes. It's running the pre-configuration and restoring an ignition gateway backup into each one of these systems. So as Randy said, 10 more days, this presentation will be done, but this is all happening live and you guys are actually seeing all of the different commands that are a part of the automation running in real time in the cloud.

29:17
Randy Rausch: Sure. So yeah, creating the secrets for the gateway network, going out and requesting certificates from a certificate authority, creating the cloud, file, cloud file shares, setting up automated backups, building.

29:34
James Burnand: Load balancer.

29:35
Randy Rausch: What's that?

29:37
James Burnand: Load balancer, the firewall rules.

29:37
Randy Rausch: Yeah.

29:37
James Burnand: There's a lot involved with having this be set up in a complete way. And that's really, the ability to do that without clicking more than one command is pretty neat. And I think what you can see on the right there is they're starting to show up now in the cluster.

29:52
Randy Rausch: Yep. So this is infrastructure live review. The resources are being spun up and created. When they turn blue, it's going through their net. When they turn blue, that pod is now running, in which case Ignition can start to boot. So it takes a little bit of time for Ignition to boot. Once it boots again, we've already configured the databases, we've configured the certificates, we've configured our MQTT broker. So when it comes online and fully boots up, it should start broadcasting to our broker. Once the broker sees it, our application should, we should see a couple of more gateways start to come online. So, what am I forgetting in stuff that we've automated?

30:30
James Burnand: Well, while we're waiting, does anyone have any questions about what they're seeing so far?

30:37
Randy Rausch: Yeah, the gist of it. And depending on how you want to architect it, if you want to have one main enterprise broker for all of them, or if you wanna have tiered brokers or staging, depending on what you want to deliver, depending on where they are, you could have load balancers in front of sets. In this case, we're having a load balancer in front of all of them, but depending on the design application that you need, you would spec it out that way.

31:00
Audience Member 1: So are, oh, so are all these pods being deployed in different availability, AZs right now, or?

31:07
Randy Rausch: Yeah, so as part of the automation, you can specify where you want them to be. For simplicity of demo, we've deployed, in this case, they happen to be all in the same availability zone, but for many of our enterprise customers, not only different availability zones, but we'll deploy in different regions to do the same sort of thing.

31:23
James Burnand: Yeah, and so these applications, the compute is in a single availability zone, but we use zone-redundant storage. So should there be an AZ failure, it'll actually, the healing process you saw would actually move to the next availability zone. So you would see, for an entire building being gone, you would see less time than what it would take to reboot a Windows machine.

31:46
Audience Member 2: Quick question about using HiveMQ here, or using MQTT broker. Is that because, I mean, would the Ignition gateway network be something feasible in architecture like this, or is that kind of not possible to shuttle tags?

32:03
Randy Rausch: Yeah, we're actually deploying the gateway network and creating the gateway network certificates as part of speeding up every one of these.

32:11
James Burnand: Okay. Yeah, we used MQTT for this, primarily because of its common use inside of an enterprise application, but also it was a nice way for us to be able to capture a handful of statistics about each one of these gateways to be able to display it dynamically, and also when it goes offline. So that was.

32:28
Randy Rausch: I'm gonna jump in right. I ran in the background an update script, and what we said is for the first 10 gateways, I wanna upgrade them from version 42 to version 43 of ignition. So we kicked that off in the background. It realized, "Hey, these are different. It has to spin down those particular containers." It then spins up a new container with version 8.43. It restores the backups and configuration and connections. So they should start to come online, and you'll see if we did it right, the version numbers should change on that top row representing upgrading our fleet of ignition gateways. So.

33:01
James Burnand: Yeah, you can see a couple of them have already started to show up, and as they're booting up and reconnecting to MQTT, they will show up as well. So from an upgrade perspective, this would be, if you're thinking about it from an application perspective, you can upgrade the version of Ignition, but that doesn't necessarily mean that your application is perfect. So what you wanna do is you wanna perform that upgrade and then do your unit testing on the application function. And should something go wrong, what would we do next, Randy?

33:28
Randy Rausch: Well, if something were to go wrong, we would use our rollback and downgrade those particular ignition gateways.

33:36
James Burnand: So we're gonna wait until they're all up before we go ahead and downgrade them, but that conceptually allows for you to have that ability to perform those tests in a way that you're reducing the risk of what would happen if there was an application problem.

33:53
Audience Member 1: So some clients that we may deal with, they're gonna have limited IT staff for ability to understand things, but they may have an application space where they actually want it managed as infrastructure. So when we're talking about getting to the edge layer into the cloud layer, does that mean opening a VPN tunnel if we're not trying to open the ports? And if we are doing something like that, do we need to worry about attack vectors like DDoS, how to respond to that?

34:15
Randy Rausch: Yeah, really good question. I understood it right. If you're dealing with something on-site as well on the cloud, do you need a VPN between them? Is that what you're asking?

34:24
Audience Member 1: Yeah, essentially for clients that have data that's existing at the edge layer, right? They wanna use a cloud-based service. So they need some way to get it up into a cloud endpoint that's publicly accessible. How do we handle that from a security perspective, from the architecture?

34:40
Randy Rausch: Great question. I'm assuming you are not coming from a place that has express routes and private links between your factories and your cloud. If you don't have that, a couple of ways to do it. VPN is one way. You can have your edge gateways VPN into your cloud and keep that private. In some cases though, like if you're using encrypted MQTT and that's what you have to send up to an enterprise broker that's secured well on the cloud, that may be sufficient for securely getting your information, depending on the...

35:10
James Burnand: We actually have had some clients, enterprise clients, that prefer public SSL encrypted data versus a VPN tunnel 'cause a VPN tunnel allows for potential traffic to come back in. So there's obviously a variety of different ways to handle that. A lot of the smaller companies will end up taking on SD-WAN projects and then terminating at a VPN endpoint in the cloud as well. So there's many different ways to solve for that. But to what Randy said, being able to have kind of a public facing MQTT SSL encrypted and then secured authenticated broker, we found to be a very well accepted posture by a lot of companies.

35:50
Randy Rausch: Took us a little over two minutes, but we did get to version 43 and we discovered, oops, we forgot to test. Let's roll back to where we were. So in this case, we're now rolling back to the previous version, which with ignition actually requires deleting the stateful set and bringing that back up to downgrade. Again, it upgrades some of your project files. But you'll see that happen in the background. Is there another one?

36:16
Audience Member 3: I know we've talked about this a lot, but what kind of feedback are you getting from customers that let's say are fairly progressive in their security posture and might have full blown Paramodel in terms of DMZs and proxies all over the place and whatnot in terms of being able to sit there and eliminate that and ultimately eliminate those operating costs from operations?

36:46
James Burnand: I would say, so you're talking about like the, I'll say the like Rockwell Cisco complex. Yeah. I forget what the name of it is, but so there's two schools of thought here. There's the, we'll call them like network layer fundamentalists who never want to kind of flatten any of that out, but to your point, there's a lot of costs that's involved in management and effort and complexity in those networks. And specifically, if you look at like how complex firewall rules get when you have three or four layers of network deep between one of your control assets and your demilitarized zone or your transfer network.

37:26
James Burnand: So we are seeing that there is a, especially as PLCs get more secure and able to exist on more IT centric and friendly networks where you're seeing OPC UA servers on board on the controllers. We are seeing a flattening of those networks. I wouldn't say it's moving terribly fast though. So it does take a little while. As Randy mentioned, one of the things that's happening here is you can't actually downgrade from 8.1.43 to 42 without deleting the hard drive in this case. So it is taking a couple of extra steps as a part of the downgrading process, which again is fully automated, but it does take a little bit of time for those resources to reprovision.

38:08
Audience Member 3: For the deployments, is there a way to, is the way you control or how something deploys, can that be changed at an individual level? Say a plant wants to change how their deployment is, or does that have to go through the centralized, like what managed the clusters deployment?

38:27
Randy Rausch: Yeah, good question. So if you are trying to centrally manage a lot of things at scale, central management is good, but you can coordinate with each individual site on when their update happens, but you would trigger that centrally. You don't have to do everything all at once, but you can.

38:44
Audience Member 3: Is there a way to have it so a local administrator can manage a deployment without having to contact the main controller that manages everyone else's deployments?

38:57
Randy Rausch: There's always a way. Is that how you really wanna do it? If someone's managing it, why do you need another manager locally? But certainly you can set it up to do on a pull basis if you prefer.

39:12
James Burnand: Yeah, typically like adding or changing services is a very fast process. So really what you saw executed here was we made changes to what Pulumi was expecting to see, and Pulumi took care of all the mechanics of actually doing that. So you can imagine if I needed an additional gateway or if I needed to have, I needed another broker or a different database or whatever the case may be, you simply need to add that into the deployment pipeline into the Pulumi configuration, and it takes care of doing that for you. So that's typically handled by whoever kind of owns that infrastructure, but the process is not, I know the process for some companies to deploy like a new virtual machine is like months of time. This is not that.

40:02
Audience Member 4: Great presentation. So you mentioned sort of at the beginning that there were potential applications that you wouldn't recommend be hosted in the cloud. I guess could you maybe elaborate what those might be at this point, and would they be something that you could instead use hybrid cloud for?

40:16
James Burnand: Yeah, so anything you would want to do on-prem hybrid cloud is a viable option because it literally is running on-prem the same as if you were running it on a bare metal server. So that is kind of interchangeable in terms of those applications. Really, it comes back to latency and user experience. So if you have a ton of transactional data going back and forth, or if this is the only thing you can do to open and close a valve and there's no other backup way to do that, you probably don't want that in the cloud. You don't wanna tolerate that up there. That's just a bad idea. So what I do think, and this is me and what I think I see happening in the future, and what I see on the enterprise application side of things is that the reliability of the connectivity for enterprises between their factories and their public cloud instances has improved drastically over the last several years. So you are finding things like warehouse management systems, ERP systems. There's a lot of time-critical things that are not living in the four walls of a building anymore. I think there is gonna be some applications that today we're not comfortable with that will maybe in 10 years from now, hopefully in my career, will make sense to be deployed in that way. And I think hybrid cloud is maybe a step in the middle.

41:32
Randy Rausch: Yeah, and you wanna take that holistic view, right? What is it you are trying to accomplish? What are the risks? Where are your points of failure? What does your overall design consider? WANs are getting very reliable and you can have redundant, triple, quadruple, whatever you need to do if that's an issue, if that's important to what you're doing, if you can tolerate the latency, you can design for that. Lots of different ways to do it, but take that holistic approach.

41:57
Kyle Van Eenennaam: Thanks so much, guys. That is gonna wrap up this presentation. Everyone, we are about out of time. Let's get a big round of applause.