The Relationship between Profits, Best Practices, and Risk in Automation Projects - Metasys N2

Best Practices

Managing Risk for Integration with Legacy Field Equipment

When a project is an upgrade, an expansion or retrofit you will be working with equipment selected, designed and installed by others. Perhaps more than one other company, because you might not be doing the 1st upgrade. This, quite obviously presents some major risks. There are a few approaches to managing this risk. Many of the risks are well known and contractors have tons of experience at estimating / allowing for it / writing change orders.

But what about Building Automation Systems or Industrial Controls Systems upgrades? In the near past this was not really an issue because closed systems meant there was little chance one vendor could replace the building level controllers of another vendor. The floor level networks were all closed. So an upgrade often meant upgrading everything – including a large amount of the installed equipment. Now thanks to protocol gateway products this economic opportunity is available to competing vendors and the game is on, as they say.

If you manage or estimate or sell upgrades and expansions for Building Automation Systems and Industrial Control Systems then you need to be aware of the Best Practices and risks and how to manage them. This article discusses these issues. It concludes by offering specific guidance for professionals who intend to upgrade which re-use existing trunks of devices by Metasys by Johnson Controls.

Three Fundamental Approaches to Managing the Risk

1. Investigate, mitigate and manage risk. Adopt Best Practices
2. Don’t accept Risk
3. Take a chance

Investigate, Mitigate and Manage Risk

Investigate, mitigate and manage risk – it all sounds so sensible and obvious. However, there are two significant drawbacks to this method – When it gets done and who pays for it.

If the customer does not perform the investigation himself and includes the results in the bid package then you have to do the work yourself. You can build the cost into your estimate. Performing the activity should allow you to provide a reduced price since managed and mitigated risk doesn’t require as much contingency. If you don’t get the job you are out of pocket. The one bright spot is that the other bidders face the same dilemma. (Perhaps you can earn brownie points with the customer by discussing risk mitigation with them. It will make you look more professional.)

Even if you don’t get a chance to inspect the site properly there are things you can ask for and information you can ask for that will give you some very strong indications of the size of the risk. Completeness of drawings, drawing age, model numbers indicate age and obsolescence, program files, screen shots, equipment lists, label files, operating procedures files, maintenance manuals and vendor documentation for field equipment. If a drawing has no changes since 1993 you can guess they are out of date. If you can’t get copies of the programming files, or if the copy you have is dated 1993 … you get the picture.

Don’t Accept Risk

You can contract out of risk provided that you know what to contract out of. Experience is a good basis for making a decision on what risks you face but it is not foolproof. There are times when we don’t know what we don’t know :) We would suggest that when you adopt a new technology, when you change your project execution style and when you do what don’t have experience with, then those are good times to do some investigation before you contract out of the usual suspects.

Is it possible to write a specification that can pre-define all the risks? Instead of investigating the quality of the existing installation you could say on your bid “This price subject to the following conditions… That all Metasys by Johnson Controls trunks currently meet the specifications and installation guidelines of the manufacturer…” Your customer may not want to accept that responsibility. They might suggest you get real. They may say “…but these are 19 year old cables, touched by many service providers over the years… I can’t guarantee them…”

Chipkin Automation Systems is developing a specification that you could include in your contract. The developed document will be specific to Metasys.

Take a Chance

Presumably you will be looking for a new job in a new industry soon. Good luck.

Key Risk Items When Upgrading Control Systems

Quality of the Existing Infrastructure

Are data trunks that connect field elements in spec? Are they overloaded? It is risky to assume the existing system is working perfectly – users may have learned to tolerate all the failures. If your job requires you to work with existing trunks you should:

1. Validate the trunks electrically

I.e. Measure voltages on a RS485 trunk. A set of rules or specification or even a test device can tell you whether the trunk is operating at the correct voltages. This should be done with a scope or if you are using a meter it should have a very fast response time and ideally have a latching function so you can note the maxima and minima. Short duration voltage events can affect data transmission significantly.

2. Validate the trunk against your project criteria

How many devices to plan to have on the trunk after the project? Is this within the operating limits of your technology choices?

What are your maximum and minimum response times? They may be significantly different from when the original system was installed and perhaps you cannot achieve your bandwidth requirements without breaking a trunk in two.

Can the upgrade system still cope with single point of failure inherent in serial trunks?

Ground potentials between the new building wing and the existing building.

On Ethernet systems you need to ensure you do not often exceed 67% of the total bandwidth. You should measure the existing usage and consider your additional load.

3. Validate the trunk against the technologies’ specification

These are the kind of things you need to evaluate.

Max then for RS485 is 4000 ft at 19200 baud. Do you intend pushing the speed up?

Unit load on RS485. System can handle a max of 32 full load 485 devices.

Are you adding 4 wire devices to a 2 wire trunk?

Does the installed trunk have a common conductor? You may need it for some devices if you are adding devices to a trunk.

Are parity settings of new devices compatible with parity settings used on existing devices?

Command Contention Complicates Automation System Upgrades

If your upgrade project requires that you leave an existing building level controller in place and that your new controller to the system, there are several risks you need to manage.

When the new system sends a command or changes a set point the change can be overwritten by the old system. It’s possible that your command will be overwritten by logic in the building level controller or even logic in a field devices such as a DX9100.

Without inspecting the logic of the field device or the building level controller you cannot tell if this can happen.

It’s also possible that these ‘overrides’ from the existing system can be manually triggered by a user sending data from a GUI / Operator Workstation.

Where systems protect humans and equipment you should complete this investigation before making the system live. Learning the hard way is often learning the most expensive and reputation destroying way.

Often you will need to hire a consultant to make this evaluation for you since you and your team don’t have expertise in the legacy system you are upgrading. Allow some budget for this. This same consultant can also validate the legacy drawings you’ve been given against what is actually being controlled and used.

Red lines show sources of command contention.

A=Field Controller logic and schedules overwrites outputs / setpoints

B=Legacy Building Level Controller logic and schedules overwrites outputs / setpoints

C=Legacy system operator work station sends schedules and commands

D=New Operator stations send commands and schedules direct to field devices

E=New Building Level controllers send commands and schedules direct to field devices

Valid Input Information

Equipment list. The ultimate project reference for how items are tagged, named and known.

Valid backups of controller logic and documentation files.

Valid backup of work station / operator interface – screens, trends, controls screens, points being alarmed.

Control System Operations Manual.

Control System Drawings.

System to System Block Diagram – The automation system might interact with other systems such as the fire, evacuation, reservations, scheduling, security systems. These interfaces need to be well understood.

Bug List. The existing system may be operating incorrectly. You should not be reproducing controls that do not work well.

Execution Style

Some ways of programming minimize risk. Use of subroutines, use of templates and of macro’s reduce risk.

A huge and common problem on projects that relate to the so called Equipment List – the list of all devices, their names, tags, labels… As you work on a project and do programming you have to name and number devices, allocate addresses, make objects and variables and name them, do floor plans, place animated objects on floor plans, make trends and logs, alarms… To do this work you have to refer to the Equipment List (or Drawings). How often does this change? Often enough on some projects to ensure that a significant portion of the project effort is just keeping up with the changes.

Team Building

If your project involves replacing another vendor’s building level controllers with your own, and re-using installed field level devices then you may need some extra help.

Does your team have all the skills required to evaluate the existing system and work with it during the changeover?

Until your own team has enough experience with the other vendor’s equipment you may need a specialist to provide the help and mentor a member of your own team on this foreign technology. For example, say you were doing an upgrade on a Metasys by Johnson Controls system. Do you have a team member who knows how to inspect the logic inside a Metasys controller? Do you have a team member who knows how to evaluate the existing data trunks? Do you have the software tools to open Metasys programming and label files?

Your purpose is to develop those skills within your own team to reduce cost in the long run. This can be achieved by having a consultant help with a specific project and at the same time transfer some of those skills to your own team.

There are times when it is extremely beneficial to pay for outside help. Probably the best time, is the period where you are developing your bid because then you can draw on the consultant’s experience to not only solve problems but evaluate the quality of the system you are upgrading and the quality of the documentation and programming that you may have to work with. The best practice is to use the consultant as a mentor to one of your own team members and make Technology Transfer part of their scope of work.

It also means that perhaps the new field tech you hired can be re-purposed. They worked for another vendor before this job, didn’t they? Instead of making your main focus getting the new hires up to speed on your own technology, use them for these legacy upgrade projects. Hire wisely. Compliment your team by drawing on the expertise of your vendors.

Be Properly Equipped

You may need to purchase the interface modules, connectors and software required to program, document the system you are upgrading. You can’t always expect the customer to have or provide these. If you use an outside consultant to evaluate the existing system you can ensure that they have all the tools to open all the system files, export labels and documentation, edit existing controller logic etc. The other route is to spend some money and purchase them for your own team. You are not only investing in the productivity of our own team, you are also making a statement to your customer about your professionalism.

Specific Advice for Those Doing Upgrade to Legacy Systems with Metasys by Johnson Controls

Step 0 – Introduction

These notes are reproduced with permission of the S4 Group. Chipkin Automation and the S4 Group work closely together to educate integrators on these issues.

Step One – Investigate the Legacy N2 Bus

You will need any or all of the following tools. The more tools you have at your disposal the greater the accuracy of your information:

• The ComBus Quick Tester – Visit the S4 Group’s website for more information of this Tool
• Johnson Controls Utilities – Often, these utilities are located on the Operator Workstation located at the site.
• HVACPRO
• GX-9100
• A digital volt meter
• Site Plans / Metasys Network Schematics

When the integration is completed the N2 bus will be driven harder than it has ever been driven before, especially when you are in a co-existence strategy with the legacy head end. That will bring out any latent problems in the bus installation or N2 devices.

Step Two – Locate and Copy N2 Configuration Files

All S4 Open appliances use N2 Device Type Templates to prove point-mapping for Metasys N2 devices. The S4 Open Appliance is capable of discovering and scanning for points on N2 devices that do not have templates, however, the points that are published upstream will not have the original user defined names. Also, not all of the points discovered may be relevant to the actual application loaded on the device. Using N2 Device Type Templates generated from the original Metasys configuration files ensures a smooth transition for the customer and the integrator.

You will need the following tools:

• Access to the legacy Metasys Operator Workstation
• A USB Thumb Drive or Other Storage Medium

The files can usually be found in the following directories:

• C:/WINPRO/ – Default directory of HVAC PRO for Windows
• C:/HVACPRO/ – Default directory of HVAC PRO for Dos
• C:/WGX9100/ – Default directory of GX-9100 for Windows
• C:/GX9100/ – Default directory of GX-9100 for Dos
• You can also search the local drives for files using Microsoft Windows search

The files have the following extensions: (Files marked with a * contain the most relevant information)

System 9100 Devices

• DX-9100
• Devices on the DX-9100 “XT” Bus
• Files can be found in the GX-9100 directory

LCP / DR-9100 Devices

• Files can be found in the HVACPRO directory
• Created by LCPCFG.EXE

Most other N2 Open Devices

• Files can be found in the HVACPRO Directory

ASC – Application Specific Controller

TC – TC-9100

VMA – Variable Air Modular Assembly

Instructions

• Copy the files that are relevant to your project to the storage device.
• These files will serve the basis for generating N2 Device Type Templates that will be used in the N2 Router to provide appropriate point mapping.

Step Three – Investigate Supervisory Equipment

This section will guide you through investigating legacy supervisory controllers, operator workstations and application data servers.

Step Four – Plan and Budget the Project

You will need the following resources:

• Completed worksheets from the previous sections
• An understanding of the goals for your project

Complete the Planning and Budgeting Worksheet.

Step Five – Discuss Results with the Facility Owner

Collect all the information that has been gathered and present the facility owner with a comprehensive overview of:

• The Current Situation
• The Work that Needs to be Done to Meet the Project Requirements
• An Accurate Estimate of the Total Time and Cost to Complete the Project

Step Six – Implement Solution