You go in for your annual physical, and for about twenty minutes, a doctor checks your blood pressure, listens to your heart, maybe orders a blood panel. Between that visit and the next one, twelve months pass in which nobody is watching anything. If something starts going wrong in month four, you won’t find out until month twelve, or until it hurts enough that you show up on your own. This is, when you say it out loud, a strange way to catch problems early. It’s also, until very recently, more or less how a 32,000-mile freight rail network got inspected: a person, a truck, and a set of eyes, walking or driving the same stretch of track on a fixed schedule, twice a week, looking for the kind of wear that doesn’t announce itself.
Union Pacific, the largest freight railroad in North America, has spent the last several years replacing large parts of that model with something closer to continuous monitoring: a network of AI-powered cameras mounted on inspection trucks and roughly 7,000 wayside sensors planted along the track itself, generating more than 16 million data points a day on the condition of wheels, bearings, brakes, and rail geometry. The pitch is the same one behind a smartwatch that flags an irregular heartbeat before you’d ever feel it: catch the drift before it becomes the failure. The complication, and it’s a real one, is that the regulatory change making this possible at scale is also the thing the people who do the inspecting are the most angry about.
What Problem Was Union Pacific Actually Trying to Solve?
Rail rolling stock and rail infrastructure fail in ways that are individually rare and collectively expensive: a wheel bearing overheats and seizes, a stretch of rail develops an internal crack under repeated stress, a coupler wears past tolerance. Twice-weekly visual inspection, done by a person walking the line or driving a hy-rail truck, catches a lot of this. It doesn’t catch defects that develop and worsen between visits, and it doesn’t scale well against a network that spans 23 states. Union Pacific’s own numbers illustrate the gap: since it began networking hotbox and acoustic bearing detectors at scale, the railroad reports a 75% reduction in mechanical derailments over the past decade, and a 28% decline in track-related derailments over the same period. Those are Union Pacific’s self-reported figures, not independently audited ones, and the railroad doesn’t break out how much of that improvement is attributable to sensors specifically versus other capital investment in the same period. Still, the direction and the order of magnitude line up with what the rest of the freight industry has reported from similar detector buildouts, which is a reasonable, if not airtight, form of corroboration.
What Did Union Pacific Actually Build, and How Does It Work?
Two systems, running in parallel. The first is the wayside detector network: roughly 7,000 fixed trackside units using thermal, acoustic, and vibration sensing to monitor rolling stock as trains pass over them at speed, no truck or person required. Union Pacific has connected about 4,400 of its hotbox detectors, the ones that read wheel and bearing temperature, into a single systemwide network, which lets machine learning models look at trend lines across a bearing’s whole service history rather than a single snapshot. Combining that temperature trend with acoustic bearing data, the railroad says it can flag a bearing at risk of overheating roughly three months before it would cross a failure threshold.
The second system, Machine Vision, is newer and covers the infrastructure side rather than the rolling stock: high-resolution camera arrays and sensor packages mounted on hy-rail trucks, paired with AI software that scans track geometry imagery for the kind of subtle, gradual deterioration a human eye tends to miss between visits. Union Pacific frames the goal explicitly as forecasting, not just detection: predicting where a stretch of track will need intervention months out, so repair crews get scheduled proactively instead of reactively.
Sensors don’t replace judgment, they change when judgment gets applied.
Here’s the obvious objection, and it’s worth saying out loud rather than skating past it: none of this actually removes the need for a human to eventually look at a defect and decide what to do about it. What it changes is when that human gets pulled in, and how much of the track that human’s attention has to cover cold, with no advance warning of where to look.
Manual Visual Inspection vs. AI-Powered Continuous Monitoring
| Manual Visual Inspection | AI-Powered Continuous Monitoring | |
|---|---|---|
| Frequency | Fixed schedule, typically twice weekly | Continuous, every train pass or truck run |
| Coverage per pass | Limited by human eyesight, walking or driving speed | High-resolution imagery + multi-sensor fusion at track speed |
| Lead time on defects | Caught if visible at time of inspection | Trend-based, up to ~3 months advance warning on bearing failure |
| Scalability | Bound by inspector headcount | Bound by sensor/camera network density |
| What it can’t do | Misses gradual change between visits | Doesn’t replace on-the-ground judgment calls |
| Regulatory status | Baseline requirement (FRA) | Requires waiver to substitute for manual checks |
What Are the Verified Results, Separate From the Self-Reported Ones?
The independently verifiable part of this story isn’t a derailment statistic, it’s a regulatory decision. On December 5, 2025, the Federal Railroad Administration approved a five-year waiver expanding railroads’ ability to use automated track inspection (ATI) in place of some manual checks, a request the industry, through the Association of American Railroads, had been pushing for years and that a previous FRA administration had declined to act on. The waiver lets railroads reduce manual visual track inspections from twice weekly to once weekly, provided monthly automated inspections and additional reporting requirements are met. That’s a real, documented, third-party decision, not a vendor claim, and it’s the actual mechanism by which AI-based inspection moves from “supplementary tool” to “partial substitute for a federally mandated safety check.”
The derailment-reduction numbers, by contrast, are Union Pacific’s own, reported in its corporate communications rather than an independent regulator’s dataset. In my experience covering enterprise AI rollouts, self-reported operational metrics from the company running the system are usually directionally honest but rarely audited to the standard a regulator’s numbers are, and this is a good example of exactly that gap: real, plausible, consistent with broader industry data, but not independently verified in the way the FRA waiver decision is.
Why Are Rail Unions Fighting the Regulatory Change That Makes This Scale?
This is the part of the story that a purely celebratory writeup would skip, and it shouldn’t be skipped. SMART-TD, along with the Teamsters-affiliated Brotherhood of Maintenance of Way Employes Division and Brotherhood of Locomotive Engineers and Trainmen, formally opposed the AAR’s waiver request. SMART-TD’s national safety and legislative director, Jared Cassity, said the waiver “does nothing to improve safety” and argued it “doubles the risk for workers, communities, and the national rail network, while padding corporate operating ratios.” That’s not a vague objection to automation in the abstract. It’s a specific claim that cutting manual inspection frequency in half, even with monthly automated checks as a backstop, trades a known safety practice for a cost reduction dressed up as a technology upgrade.
Notably, Cassity didn’t reject automated inspection outright. He said the union supports ATI when it’s used alongside visual inspections rather than instead of them, calling that combination “a sensible and pragmatic approach.” That’s a meaningfully different position from “ban the cameras,” and it’s worth taking seriously: the disagreement isn’t really about whether machine vision works, it’s about whether it’s good enough, yet, to stand in for a person rather than stand next to one. This debate is unfolding against a rail industry still under heightened public and regulatory scrutiny following high-profile derailments in recent years, which raises the stakes on getting the substitution question right rather than fast.
AI Suite and Open-Source Implementation
Union Pacific’s stack here is best understood as two layers stitched together rather than one platform. The bottom layer is sensing infrastructure: thousands of fixed wayside detectors (hotbox, acoustic bearing, and related sensor types) feeding a systemwide telemetry network, plus vehicle-mounted camera and sensor arrays for the mobile track-geometry side. The top layer is the analytics: machine learning models trained to spot trend lines in bearing temperature and acoustic signatures that predict failure months out, and computer vision models trained to flag track geometry anomalies from imagery. Union Pacific hasn’t published the specific model architectures or cloud vendor behind Machine Vision, which is common for safety-adjacent systems in a heavily regulated industry, but the pattern (sensor fusion feeding purpose-built ML models, rather than a single off-the-shelf platform) matches what peer railroads and vendors like Wabtec’s KinetiX inspection line, now running in 400-plus installations worldwide across freight, transit, and heavy-haul rail, have described publicly.
An enterprise wanting to replicate the underlying capability, not the exact system, doesn’t need to start from a rail-specific vendor. The core computer vision problem, detecting surface and structural defects from imagery at speed, has a real open-source research base: published architectures like RailTrack-DaViT (a vision transformer approach to automated track defect detection) and YOLO-family object detection models adapted for rail surface defects are documented in peer-reviewed work, and both build on widely available frameworks (PyTorch, OpenCV) rather than proprietary tooling. The sensor-fusion side, correlating time-series data from multiple sensor types to predict a single failure mode, is a more generic industrial IoT and predictive-maintenance pattern, not something unique to rail.
For an enterprise building toward something similar: first, inventory what you’re already collecting from existing equipment sensors before buying anything new, most industrial operations generate more usable telemetry than they’re currently modeling. Second, start the computer vision or predictive model on a narrow, well-defined failure mode (one component type, one defect category) rather than a general-purpose inspection system, since narrow models are both easier to validate and easier to explain to a regulator or a skeptical workforce. Third, treat the output as a prioritization signal for human inspectors before treating it as a substitute for them. The honest caveat: open-source models can replicate the detection capability reasonably well on public benchmark data, but they cannot replicate Union Pacific’s decade of proprietary failure-outcome data tying sensor readings to actual field failures, which is what makes the three-month-advance-warning claim credible in the first place. Any new deployment should run in shadow mode, flagging predictions without acting on them, until its outputs have been validated against real outcomes over a full seasonal cycle. Skipping that step is how a promising pilot becomes an unreliable production system.
FAQ
Q: Does Union Pacific’s AI system replace human track inspectors?
A: Not entirely. It changes the balance between automated and manual checks: under the FRA’s December 2025 waiver, qualifying railroads can reduce manual visual inspections from twice weekly to once weekly if they run monthly automated inspections, but human inspectors and judgment remain part of the process, and rail unions specifically object to any framing that suggests otherwise.
Q: How much has AI actually reduced derailments at Union Pacific?
A: Union Pacific reports a 75% reduction in mechanical derailments and a 28% decline in track-related derailments over the past decade, coinciding with its wayside detector buildout. These are self-reported figures, not independently audited, though they’re broadly consistent with industry-wide trends reported by other Class I railroads using similar sensor networks.
Q: Why do rail unions oppose automated track inspection if it improves safety?
A: Unions like SMART-TD don’t oppose automated inspection technology itself, they support using it alongside manual checks. Their objection is specifically to reducing the frequency of required manual visual inspections in favor of automation, which they argue trades a proven safety practice for cost savings before the technology has fully earned that substitution.