Guidelines for Transportation Management Systems
7. Maintenance Program Support Services
This chapter describes the variety of support services can assist in the development of a maintenance program. The reader should use this chapter as a list of reminders to determine if any of these items are appropriate for use in the development of his or her Maintenance program. The support items considered here should be considered individually for each TMS component under consideration. Some of these items are not applicable for all components. The items include:
As defined in Chapter 1, responsive maintenance is the repair or replacement of failed equipment and its restoration to safe, normal operation. Preventive maintenance is the activity performed at regularly scheduled intervals for the upkeep of equipment. The topics covered in this chapter include how these maintenance activities may be supported by Federal funding and how they may be contracted out to industry. The supporting tools of assessment management are included as well as information on warranties. A sample contract for maintenance is also presented along with details on acceptance testing procedures that can assist in the development of a Maintenance plan.
These types of support services can provide value to any Agency that is developing a Maintenance plan. However, some of these services, such as asset management, can initially require a significant level of start-up costs and time. However, the value of contracting out components of maintenance can add significantly to the capabilities of the Agency in terms of technical expertise and extended hours of coverage. Each of these support services should be carefully evaluated by each Agency considering the trade-offs among costs, responsiveness, and impacts on staff.
One example of how the value of support from other Agencies can be applied is the effectiveness of distributed maintenance centers in Agencies with large geographical coverage. A state Agency with a low population requires the central maintenance staff to fly to remote locations in order to reset controllers where the conflict monitor has been tripped by lightning. In another state, the signal controllers are many hours drive away and an overnight trip is required. Although the responsive action of resetting the controllers is essential, there are a range technological options and local contracting alternatives that should be considered when such inefficiencies arise.
The maintenance component of TMS is expensive. For large systems, annual costs can be a few million dollars per year. In Texas, the annual ITS/ATMS budget request for Fiscal Year 1997 was $8.7M from Routine Maintenance and $5.0M from Highway Construction for rehabilitation and maintenance. For a large ITS in Northern Virginia, VDOT has contracted out the maintenance at a cost of over $2M annually. Washington has an annual combined O&M budget of $1.9M for 800 signals and 120 miles of freeway surveillance. Maryland has an operations budget of $3.5M for 375 miles of freeway surveillance and a maintenance budget of $1.0M. For the ARTIMIS system, ODOT contracts over $1.0M per year for maintenance. These are fairly large numbers and Agencies do need to analyze the options associated with contracting out maintenance.
Agencies should consider that Federal funding is available for maintenance.
The Federal-aid Eligibility Policy Guide (Ref 10) states:
Each Agency should consider a range of issues when planning whether to contract for maintenance or conduct the work in-house, including those items concerning contracting procurement issues. The decision to contract out is often made in an environment where the addition of several new members of staff in a transportation department would be considered not politically judicious.
Virtually all Agencies have some form of contracting in their maintenance programs — these contracts are most common in the repair of communications systems, as these require sporadic repairs and very specialized equipment. Given no political constraints on the employment of new staff, adequate physical space for equipment inventory, and personnel, it is likely that the costs of in-house maintenance would generally be less than outsourcing the work. Lack of overhead and no profit on the project virtually ensures that in-house maintenance is cheaper when viewed in terms of an annual budget. However, the nature of employment in government Agencies involves long-term commitments and their associated costs, including medical benefits, pension plans, vacations, and other costs. When these are taken into account, the financial balance moves in the direction of contracting out at least some of these services.
Which maintenance tasks to contract out can depend on the type of Agency in terms of the in-house skills, hours of operation, and the equipment space and facilities that are available. The in-house skills can affect the decision in that some Agencies have competently-trained staff and equipment to provide roadside maintenance of the ITS devices. In some cases, this competence results from the Agency installing the devices themselves. This approach results in a familiarity with the equipment and access procedures and enables the in-house staff to perform preventive and responsive maintenance. Operational requirements can also influence which maintenance activities to contract out. If the Agency wishes to operate the system either for extended periods or full-time, then the staff may not wish to be on-call 24 hours per day. In these circumstances, responsive and emergency operations outside office hours can be used to supplement the work performed by in-house staff.
An example of one in-house maintenance operation is provided by Denver CO (Ref 11). The Colorado Transportation Management Center (CTMS) Maintenance Division is responsible for deployment, maintenance, and development of ITS field devices. The staff includes one Electronic Engineer working as crew supervisor, one Electronic Specialist I, one Electronic Specialist II, and one Electronic Specialist Intern. Additional support is provided by two contract maintenance staff members.
The primary function of the CTMC maintenance crew is the upkeep of ITS field equipment that is controlled by the Colorado Traffic Management Center. This equipment includes variable message signs, highway advisory radio transmitters, CCTV cameras and video switchers, roadside emergency call boxes, wireless communication devices, highway traffic sensors, and fiber-optic equipment. Additional equipment is continually being added to the inventory as the ITS infrastructure grows. Although the majority of ITS devices are located north and west of Denver, equipment locations maintained by the CTMC range from Vail on I-70 to as far south as Pueblo, Wolf Creek Pass in the southwest, and north to the Wyoming border. The number of devices maintained by the CTMC is now in excess of 100.
The CTMC Maintenance Department has also been charged with deployment of field devices. Everything from installation of highway advisory radio equipment to closed circuit television cameras is handled by the CTMC crew. The CTMC has a fleet of 10 portable variable message signs at its disposal. Used for road information and in support of law enforcement DUI check points, these signs are deployed by the maintenance crew within a 100-mile radius of Denver. The CTMC maintenance fleet also includes a bucket truck with an 80-foot reach and a dedicated "network support" vehicle used to diagnose problems with CDOT's fiber-optic network.
The CTMC Maintenance Department also designs and fabricates new electronic systems used to assist the motoring public. These systems frequently make use of both fiber-optic and wireless communications devices. For example, they are working on the development of remote controlled "trailblazer" signs that utilize wireless technology. These signs are used to direct traffic flow when detours are required because of various road conditions.
Western Transportation Institute (Ref 2) cited in its study for ODOT the following deficiencies in the ODOT Maintenance Program:
The development of a series of problem definitions can assist in both the production of a Maintenance plan and the decision to contract out part of the work. Those areas that are not being adequately maintained tend to be perceived as problems. Sometimes these are functional, occurring in areas where there is little or no in-house expertise. In other situations, the problems may occur in one geographic area where a certain districts lack either expertise or adequate staff.
When making decisions about contracting, each Agency needs to look at its individual deficiencies and needs in terms of functions and geography. The advantages of contracting out maintenance include:
Another issue that the Agency needs to consider when developing a plan is the relationship between preventive and responsive maintenance activities. Doing more preventive maintenance will necessitate less responsive maintenance. There is no subjective evidence to quantify the extent of this relationship in the ITS industry. There are, however, numerous examples in asset management models, fleet operations, and the steel industry that evaluate production capacity and uptime as a function of maintenance expenditure. There are diminishing returns — i.e., beyond a certain point, the use of funds for preventive maintenance is not cost-effective. There will always be a need for responsive maintenance due to inherent equipment failures, knock downs, lightening strikes, etc. However, money spent on meeting the basic maintenance requirements specified either by the manufacturer or using the procedures outlined in this guide will reduce responsive maintenance incidents and be more cost- effective.
The FHWA and AASHTO (Ref 12) define asset management as follows:
The same reference defines a series of questions to ask when developing an asset management system. These include:
There are a number of suppliers of asset management systems whose web sites provide useful information:
Hansen - http://www.hansen.com/
Some Agencies do no preventive maintenance, considering it a luxury. This is unfortunate. Waiting for devices to fail is inefficient and leads to compromised operations. It is inefficient insofar as the damage done when devices fail can be significantly more expensive than undertaking preventive measures. Individual components will eventually fail and need to be replaced — this is, almost invariably, a more expensive option than cleaning, greasing, measuring, etc. Additionally, the responsive actions that will occur more frequently when preventive maintenance is not performed will happen at indeterminate times. Preventive maintenance can be scheduled to not disrupt traffic flow and be considered part of the workday of Agency or contractor staff. The operational effects of waiting until things break causes the failure to be noticed when the device is used. This is the worst time as the device is needed by the operator, the traffic in the area may be subject to an incident, and the staff is busy.
If an Agency determines that some component of their maintenance is to be contracted out, there are a variety of procurement options. Some of these may not be available to individual Agencies due to local restrictions on procurement procedures.
The list below covers most of the procurement options that are used for TMS. Some of these are applicable for operations and maintenance, and others are not:
Engineer-Contractor: Plans and detailed specifications are developed by an engineer. Selection of a contractor is by low-bid only.
System Manager: Plans, specifications, and system software are developed by an engineer. Selection of the equipment and installation contractor is by low-bid. The engineer provides system integration.
System Integrator: Similar to System Manager, but the contractor has the ability to procure hardware and services, usually by low-bid, on behalf of the Agency and then functions as an integrator.
Best-Value Contracting: Used where the contractor must provide some technical designs or configurations. The selection of a contractor is based on a combined technical score and price.
Design-Build (DB): A set of performance specifications
are let for bid by teams of engineers and contractors. Selection is usually
based on combined technical and price factors.
Companies that win bids for the maintenance of ITS devices are typically local electrical contractors, sometimes working with specialist subcontractors. Generally they are not experts in ITS as these opportunities are rare. Most of their work is for commercial clients on new buildings and facilities. In some cases, they may have experience with other surveillance systems or cell tower installations and this should be considered as a plus during selection. However, it is most likely that the winning bidder does not have expertise in most of the devices that they are installing. Vendor warranties on items such as DMS's are normally for periods of 1-2 years. From the Agency's perspective, this is a high-risk situation — these devices are generally unreliable and it is rare to come across a sign in the field with all pixels operating correctly.
More reliable devices such as cameras and communications electronics are also only usually warranted for only one year. Most vendors will provide an extended warranty for a price. If the warranty terms are passed through a third-party contractor, the extended warranty costs will undoubtedly get marked-up. The Agency may wish to consider purchasing the warranty directly from the hardware manufacturer and making the Agency staff responsible for managing the warranty repairs.
The record-keeping system should provide information concerning the performance measures of the various devices. The Agency should attempt to estimate the mean time between failures together with the cost to repair/replace the equipment — that information can then be used to judge whether the extended warranty is worthwhile. A simple calculation taking the mean time to repair and the number of devices should provide the annual expected number of failures; multiplying this by the mean cost to repair will provide the annual cost, and this can be compared with the quote from the vendor. However, allowance should be made for the costs associated with removing, replacing, contacting the vendor, handling and shipping, and purchase order production.
The standard warranties and service agreements that come along with the purchase of a new product are usually dated from the time that the device left the factory. Contractors tend to order equipment early in the project if the costs can be billed to the Agency. This approach allows the contractor to invoice early and become familiar with the equipment. The permitting, construction of concrete bases, and the provision of electrical power and communications can often take more than a year. Thus the situation can readily arise that the device is not fully tested on site until the warranty has expired. It is, consequently, prudent for the purchaser to time the delivery of devices such that the warranty starts when the device is installed and operational.
Extending the warranty period will affect the price, though sometimes less than anticipated. Since the contractor will use the original vendor warranty for the initial period, the contractor has few costs during this time. Additionally, some devices tend to have most of their failures during the first few weeks of operation.
Some Agencies do not encourage extended warranties, requiring instead that the construction contractors manage everything. These sorts of issues need to be considered as part of the procurement process. If necessary, a supplemental contract can be used for extended warranty.
Each Agency will need to develop its own contract and apply its own procurement procedures and the associated boilerplate text that is required.
Contract Forms and Pricing
The scope-of-work and the level-of-effort can be defined for preventive maintenance. Since the number of devices and their maintenance frequency is known and the duration of each operation can be estimated, potential bidders can provide cost estimates for preventive maintenance that have reasonably low-levels of risk. If a well-defined scope-of-work is used, the preventive maintenance can be bid as a lump-sum contract.
The responsive and emergency maintenance components of the contract need to have sufficient definition to encourage bidders to bid and be formulated to minimize their risk. One approach is to specify that a certain staffing level — e.g., defining the number of persons and hours of operation — be made permanently available to the TMC manager. This approach is suitable for large systems where the number of staff working on purely responsive operations is sufficient to justify paying them for full-time positions. However, when preventive and responsive maintenance are performed by the same crew, there are significant efficiencies in terms of reduced plant and staff. The maintenance crews can perform preventive maintenance until called upon to do responsive or emergency maintenance. If there is more than one crew in the field, the response time can be reduced by using the crew nearest the fault.
One solution to this problem is to develop an estimate of how many responsive calls are liable to be made and make the pay item per call for a specific type of device. Thus, the Agency estimates how many calls should be used in the basis for the bid. The bidder will bid per responsive call type. Using this approach, a bidder's costs can be readily calculated and the Agency can still select the low bidder on a lump-sum contract. This approach does mean that there is an administrative effort required by both the bidder and the Agency to keep track of the number and type of responsive and emergency calls and to verify these with the monthly invoices.
Note that most contractors will not bid on an ill-defined responsive maintenance contract that was lump-sum. If they did bid, there would be a very large contingency to account for their risk. Consequently, each Agency, when developing the contractual terms and scope-of-work, should consider the calculations that will be made by the bidders. Preliminary meetings between bidders and Agencies to help define each others' concerns, together with a clear scope of what is required, is more likely to lead to a successful maintenance contract.
Outreach to all bidders in the region and, in some cases nationally, will significantly assist the Agency in getting more bids. There are national electrical contractor sources that can provide lists of potential bidders by region. This can be searched on-line at http://www.thebluebook.com/.
In addition to adjacent jurisdictions, there are other external maintenance companies that should be included. These could include companies that maintain cellular towers, external lighting, and private security systems. Providing lots of detail on the potential work and a contact number to answer questions will assist in broadening the list of potential bidders.
Appendix C contains a generic contract that can be used as a basis by Agencies developing a maintenance contract scope-of-work.
Appendix D includes a generic contract based on software procurement for the maintenance of a central system. This can be used as the basis of a purchase order or contract between a software system integrator and an Agency. In addition, this appendix contains an example of a purchase order form of contract that can be used as a model.
Acceptance testing is an intrinsic part of all TMS implementations. It provides the Agency with reassurance that the TMS functions are operable and the required hardware is in place. It also provides the maintenance staff with an introduction to specific equipment in particular locations.
Acceptance Test Plans
The Agency should provide detailed acceptance test plans for all system components. Although the contractor can produce these plans, the Agency will need to conduct a detailed review of such plans to help ensure it is getting what it wants.
In the case of software acceptance tests, there should be traceability between the original requirements and the tests. At each level of the decomposition of the original requirement, more details and functions should appear. The acceptance test plan for the software should trace each individual test back through the detailed design into the original requirement.
The contractor, under supervision by the Agency or its representative, will often conduct the tests. The test results should be documented and the documentation delivered to the Agency. Any failures during the testing process must be documented and then fixed by the contractor and the test repeated and again documented.
Factory Acceptance Tests
Factory acceptance tests provide the Agency with an early useful view of products with which they may be unfamiliar. The look of a DMS, the shape of a structure, or the color of a cabinet is difficult to specify and there are many such items that the maintenance staff may wish to review during the construction. Objections to an item that was not specified, after the products have been produced and delivered to the site, generally cannot be accommodated.
During factory acceptance testing, the manufacturer can demonstrate their construction techniques and quality-control procedures.
Communications System Acceptance Tests
Many ITS implementations are late and often over budget. A major reason for this is the communications system. It is usually very complex, can involve multiple jurisdictions and commercial suppliers, and often involves implementation of new technology.
The test of the communications system should ensure that communications are functioning as specified in the statement of work. The contractor should conduct a series of communications tests designed to prove that the system can properly carry and pass through all the messages required to operate the system. These communications tests often include end-to-end bit error rate tests, data and message integrity tests, and other assessments demonstrating proper operation.
A Test document, describing the means and methodology of the communication tests, should be presented to the Agency some time in advance of the start of testing. Testing should not begin until the Agency has approved the testing plan. As before, all testing procedures should be documented while the testing takes place.
If the communications system consists of multiple "legs" that are controlled by disparate institutions, it is advisable to split the acceptance test into components. For example:
Thus there is a long chain of differing responsibilities and the maximum opportunity for finger pointing when the data does not arrive. This problem can be minimized by performing separate tests on each link in the chain.
Stand-Alone Site Tests
Individual test plans should be prepared for each component and be performed at each site in the presence of the Agency, or its representative.
The stand-alone test plan shall determine that each element at the individual sites is operating correctly in isolation. For example, all DMS suppliers provide software driving their signs that can be run on a laptop. Running through their test program on-site can test the sign in isolation from the communications system. Similar approaches can be used for video and surveillance components.
Integration test plans should also be prepared. These plans determine that individual system elements at each site are operating correctly in an integrated manner.
Integration tests occur following successful demonstrations of the stand-alone tests.
There are multiple points of potential failure in most systems and an effective acceptance test plan for the integrated system will need to consider each element.
The acceptance testing is usually linked to payment milestones. For example, the installation of a DMS might have milestone payments split into three components, as follows:
It should be realized that virtually no component of a TMS can be repaired by anybody other than the original manufacturer or a qualified distributor. The process of responsive maintenance typically involves replacements of boards and modules.
The provision of adequate spares (usually 10 percent of the total installed) and locating them in an easily accessible manner will have the biggest impact on reducing down time.
The maintenance of a TMS typically requires the kinds of support tools that a maintenance department is likely to have on hand. These include trucks, cherry pickers, sign boards, back hoes, etc. In addition, more specialized equipment specific to the electrical nature of the work are likely to be needed. This equipment can include:
For responsive maintenance, these types of devices are used to determine what and where the problems may be. For the communication elements, for example, the OTDR is used to determine where on a communications network any break in the connection may have appeared. Following this determination, the traffic control, back hoes, and fiber fusion equipment is needed. In addition, when the fiber is fixed, packages of bonding material and various resins are used to replace the cover to the fiber. Following this, the cable is retested and the holes filled in. This process needs to take place either as a contracted activity or as an operation performed by the Agency's staff. Regardless, substantial equipment, expertise, manpower, and materials are required.