The drone light show is part of an episode of “The Secret of Skinwalker Ranch”, a TV series exploring the mysteries of a location with an extensive history of unexplained phenomena, such as UFO sightings, crop circles, and unexplained animal mutilations. The show’s team of scientists and experts utilize leading technology, such as drones, to study and investigate these occurrences. The new episode, “Who’s Your Data,” premiered this past Tuesday on The History Channel.
The episode focused on deploying a swarm of drones to discover aerial anomalies. The show’s team invited Sky Elements Drone Shows to conduct a safer and more thorough scan using Drone Show Technology to collect data and research the anomaly in various areas in the ranch.
The company deployed their drones, normally used for light shows, to disperse over the Ranch’s anomaly areas to watch illuminated formations against the night sky, making it easier to spot any changes in the pre-programmed patterns and locate the anomalies. The team from Sky Elements Drone Shows flew the illuminated swarm of 200 drones in a massive grid formation over the anomaly areas, allowing them to monitor the drones’ exact position above the target areas.
“Our participation in the latest episode of “The Secret of Skinwalker Ranch” has been an exciting opportunity for us to showcase the potential of Drone Show Technology from SPH Engineering. Through this collaboration, we were able to conduct a unique and innovative experiment using our drones to investigate the enigmatic anomaly areas on the ranch. The use of our cutting-edge software allowed for a safer and more comprehensive scan of the target areas, providing valuable data and insights to aid in the scientific exploration of Skinwalker Ranch. We believe that the successful integration of drone technology in this experiment highlights the immense potential of this technology in scientific research. We are proud to have been a part of this groundbreaking project with “The Secret of Skinwalker Ranch” team,” said Preston Ward, Sky Elements Drone Shows’ General Counsel and Chief Pilot.
“Sky Elements Drone Shows has once again demonstrated the potential of drone technology in entertainment and scientific research,” said Alex Levandovskiy, head of Drone Show Technology division at SPH Engineering. “We are thrilled to be a part of this unique experiment on The Secret of Skinwalker Ranch, which showcases the capabilities of our Drone Show Technology in supporting cutting-edge research and exploration. This collaboration with Sky Elements Drone Shows is a testament to our commitment to providing comprehensive drone software solutions that can be customized for any occasion or event.”
SPH Engineering’s Drone Show Technology is a comprehensive drone show software solution enabling the creation of complex 3D animations, the real-time synchronization of drones, and management of the entire show from a single interface. It is compatible with a wide variety of drones and lets users create customized shows for any occasion.
Read more:
Ian attended Dominican University of California, where he received a BA in English in 2019. With a lifelong passion for writing and storytelling and a keen interest in technology, he is now contributing to DroneLife as a staff writer.
Miriam McNabb is the Editor-in-Chief of DRONELIFE and CEO of JobForDrones, a professional drone services marketplace, and a fascinated observer of the emerging drone industry and the regulatory environment for drones. Miriam has penned over 3,000 articles focused on the commercial drone space and is an international speaker and recognized figure in the industry. Miriam has a degree from the University of Chicago and over 20 years of experience in high tech sales and marketing for new technologies. For drone industry consulting or writing, Email Miriam.
Labor, machines and materials—the three critical elements of any infrastructure construction effort—have begun to realign in a way that will forever change the way infrastructure is built…and likely operated.
While some might be skeptical, the future of autonomous vehicles will, at least in part, likely draw from lessons learned in the physical construction of roads, bridges, transit systems and other infrastructure.
Autonomous machines are already working in coordination to build dams and large-scale land developments such as solar and wind sites. Synchronized self-driving trucks are the norm at many mining sites around the world. While there are fewer examples on road, rail and bridge projects, robots are tying rebar for new bridge decks, performing jobsite inspections in the air and on the ground, and soon will enable transportation agencies to autonomously repair potholes. In many ways, the construction space has become a proving ground for autonomous advancement—and investors are taking notice.
The ResearchAndMarkets.com report, “Global Autonomous Construction Equipment Market Opportunities and Strategies to 2030: COVID-19 Growth and Change,” noted that the global autonomous construction equipment market was about $8.45 billion in 2020, a steady but modest increase of 3.4% since 2015. That investment is expected to double to $16.89 billion by 2025, a 14.9% compound annual growth rate, and jump another third to $26.16 billion by 2030.
Beyond the investment dollars, the report authors believe the fully autonomous construction equipment segment will be the fastest-growing piece of the autonomous puzzle, at 49.1% from 2020-2025. That’s a big shift, considering the partial/semi-autonomous market accounted for 97% of the total in 2020. One caveat to the forecast growth is that a majority of the autonomy pieces are expected to be in earthmoving and material handling for the building construction market—not actual road construction.
However, advances in semi- and fully autonomous excavators, dozers, rollers, soil compactors, skid steers and other machines by technology innovators and OEMs may rewrite those predictions. As Thomas Hansen, senior director OEM Development at Topcon Positioning Group, noted, “Those outside the construction space might be surprised how far we are and how quickly our vision for the autonomous road ahead is clearing.”
SOLID FOUNDATIONS
Much like self-driving cars, semi- or fullyautonomous construction equipment rely on a combination of cameras and sensors. Data from LiDAR, cameras, GNSS, vibration and temperature sensors, etc., combine for optimal perception and reliable environmental mapping that are critical for autonomous solutions.
“We are experiencing a great convergence of key technologies that are enabling the industry to accelerate toward both automating machines and optimizing the site management/machine workgroups,” Trimble’s Autonomous Solutions Business Area Director Finlay Wood said. “There is no single area of development that is responsible for this; rather a maturation in multiple areas including communications, advanced perception solutions, edge computing, machine control solutions, site management software and many more.”
Machine control, for instance, is a clear step toward true autonomous functionality. Solutions such as Trimble’s Autos mode are built into its Earthworks grade control software. With Autos on an excavator, the machine controls the boom and bucket to stay on grade while the operator controls the stick. More recently, the company added horizontal steering control for soil compactors, which is said to be an industry first. Trimble Earthworks Horizontal Steering Control automatically steers a soil compactor using a 3D model or compaction pass line.
Built Robotics, a vehicular automation startup out of San Francisco that develops manufacturer-agnostic systems, deployed the first U.S. commercial autonomous construction equipment—the Exosystem—in 2019, according to the firm. Installed on excavators, the Exosystem enables big machines to operate autonomously. With a high-powered computer, IMUs, GNSS, radar and 360 degree cameras, the solution is built to trench fully autonomously in various soil conditions and desired depths. Mortenson, one of the nation’s top-20 builders, developers and engineering services providers, has put the Exosystem to work on earthmoving projects in clean energy for wind and solar farms. And it’s not just the contractors that see the value. The International Union of Operating Engineers partnered with Built Robotics to begin training its members in the next-generation skills needed to operate robots at their facility in Crosby, Texas.
Hyundai Doosan Infracore, an affiliate of Hyundai Heavy Industries Group, a leading manufacturer of heavy equipment, is building on its machine guidance and machine control capabilities as well.
Specifically, the company’s Concept-X project, ongoing at the Boryeong Proving Grounds in South Korea, is focused on commercializing development for step-by-step element technologies such as machine guidance and machine control for machines such as excavators and dozers, some of which have already been released.
“In 2022, we’re developing and releasing the machine guidance and machine control technologies’ for our excavators, dozers and wheel loaders,” said Todd Roecker, director of growth initiatives at Hyundai Doosan Infracore North America.
But that’s just the beginning. Working with experts at Seoul National University, Yonsei University and Hanyang University, the Doosan team is focused on a digital construction platform for smart construction and automation—the XiteCloud platform. The XiteCloud combines information, communications and AI technologies, and will be a foundation for autonomy.
“We’re preparing for the proving of the autonomous construction solutions in real field applications with our Korean construction company partners,” Roecker added. And Doosan is not alone in the focus on a smart platform to drive future innovation for construction.
THE 3D ECOSYSTEM
An underlying element of all recent advances and a true move to autonomy is a 3D data ecosystem.
As Erol Ahmed, director of communications for Built Robotics, said, “Data is the elephant in the room to any form of autonomous operations. That, and interoperability, are what will give the power to these machines.”
Topcon’s Hansen agrees, adding: “3D is the foundation that’s going to allow the next level of autonomy in construction. Everything on the machine, all the sensors working together, will be represented in that model. It’s imperative that we as an industry build a stable growing and interoperable platform that allows all current and future product data to integrate.”
There’s still work to do. “The machines need to be aware of the environment and the environment needs to be aware of the machines,” said Jason Hallett, CEO of Digital Construction Works (DCW), a technology integrator. “Right now, for most systems, that’s not the case.”
Even machines with the most advanced technology are only aware of where they are, and, to a certain extent, the things around them thanks to sensors and GNSS.
“They aren’t equipped to adjust to changing conditions,” Hallett continued. “Unlike autonomous cars that are traveling on a mapped roadway, a construction jobsite is changing continually and can look completely different over a couple of days.”
As an example, the firm-developed DCW 4D mass haul earthworks solution optimizes earth moving operations with integrated civil planning workflows, including 4D earthwork project planning, simulation, mass haul schedules and quantities. The solution works thanks to an integration with Bentley’s Synchro, a 4D software platform that virtually and visually enables planning, scheduling and controls with Topcon’s MAGNET Project and Sitelink3D technologies, which plan and track construction site material weighing and haulage.
“The idea is to synchronize plans for mass haul with what is happening on the job to provide a near-real-time mass haul update,” Hallett said. “By synchronizing inputs and outputs, crews can make more informed and timely decisions related to schedule and resources.”
It’s that project and site focus that is also driving Trimble’s advancements within the 3D ecosystem. “It is not good enough to have a fantastic robot on a job if you can’t remotely create the work order and then ensure the machine is executing the task in an optimal manner, and then provide relevant data back to the site management solution,” Wood said. “We must be able to turn real-time data into real-time information to optimize and coordinate the workgroups of machines and the site—which goes well beyond point solutions.”
One way that will likely happen is through partnerships such as a Trimble-VayaVision alliance to make intelligent automation even smarter. The idea is to help contractors, for instance, scale automation efforts from driver assistance features to fully autonomous solutions, augmenting or replacing human control to create a safer work environment through path optimization, obstacle avoidance and situational awareness.
“We believe a robust 3D environment will drive path planning and autonomous systems in both off-road and on-road applications,” Wood continued.
Though the fact that a 3D ecosystem is not fully developed has not stopped OEMs, technology developers and contractors from pushing forth with autonomy. One of the necessary areas of advancement is the ability of multiple machines to operate together.
As Ahmed confirmed: “As much as one machine can provide great efficiency and productivity, a construction project includes different OEMs, ages of equipment, contractors, etc. Fleet communications is critical to the construction ecosystem and the future of autonomy.”
One of the most visible demonstrations of an autonomous fleet solution on roadway construction was spotlighted in Asia: Chinese construction machinery manufacturer XCMG’s autonomous road roller, developed along with Sichuan Railway Investment Group and Tsinghua University. According to XCMG, the road roller is able to operate autonomously under complex conditions with a precision of 2-3 centimeters. The road roller has various safety features built in, including a real-time monitoring system that can sound warnings, make an emergency stop, automatically enter and exit sites, and avoid obstacles.
In late 2021, XCMG and its partners demonstrated what they say could be the largest unmanned cluster paving and intelligent maintenance construction effort in the world on the more than 48-kilometer, four-lane Zhengzhou-Xixia Express. For the project, 10 XCMG driverless double-steel wheel rollers and two XCMG large width pavers used high-precision BeiDou positioning, inertial navigation, obstacle identification and multiple safety systems to pave the road, even in extreme weather and special terrain. The autonomous fleet network included AI and 5G, and a range of sensors for safety and visualization. Moving forward, the partnership will focus on R&D and implementation of smart transportation development strategies.
HIGH PERFORMANCE PROTOTYPES
As part of the autonomous vision for construction, manufacturers have initiated a number of innovative programs that have pushed the industry forward in recent years.
For many, the autonomous focus is still largely in greenspace areas or enclosed environments such as mine sites. For instance, Honda collaborated with Black & Veatch, a global engineering, procurement, consulting and construction company that specializes in infrastructure development, to test the prototype Honda Autonomous Work Vehicle (AWV) at a large-scale solar Black & Veatch energy construction project in New Mexico last year. Tasks included towing activities, and transporting construction materials, water and other supplies to pre-set destinations within the work site.
Volvo Construction Equipment introduced the Volvo LX03, autonomous concept wheel loader, and the CX01 autonomous, self-leveling asphalt compactor prototype. The compactor, in particular, is operable by remote control or entirely autonomously. The wheel loader is especially unique, as it is the first real-world example of a self-learning wheel loader, designed to make decisions, perform tasks and interact with humans. While not commercially available, engineers expect that valuable insights from the LX03 will feed into applications for today and tomorrow.
Earlier this year, Houston-based Bell Trucks America teamed with Pronto AI of San Francisco to develop what they say will be the first commercially available off-road self-driving articulated dump truck featuring Society of Automotive Engineers (SAE) Level 4 autonomy (the ability to act without any human intervention in the vast majority of situations).
The solution combines Pronto’s plug-and-play AI hardware and software with Bell’s autonomous-ready platform. While only tested in mine sites thus far, the technology is designed to safely navigate unpredictable, busy highway environments. In the near term, the self-driving solution will be tested on less congested and more predictable offroad operations, to assure all-important safety protections and operational effectiveness.
As well, Japanese construction machinery maker Komatsu has introduced bulldozers with autonomous blade control. Komatsu says the system collects data at the tracks (instead of the more common collection at the blade), thereby providing a real-time 3D picture of the ground around the machine—a potentially critical differentiator for autonomous solutions.
Entrepreneurs are also jumping into the construction space. One example is Silicon Valley-based SafeAI, a tech innovator that modifies trucks, bulldozers and loaders with AI-powered solutions to operate autonomously. SafeAI executives say they have already developed small-scale, real-world tests on skid loaders.
INDUSTRY-WIDE EXCITEMENT
The future is bright for autonomy in construction.
“I believe we as an industry are on the right track to build the best possible 3D ecosystem, but there are some other things that must happen,” Hansen said. “We have to get the cost of sensors down to make it reasonable for the construction industry to invest. As well, there will be some regulatory adjustments, some clarity on insurance conditions and, of course, continued focus on cybersecurity. But the technology is coming.”
Ahmed situated the shifting environment: “We’re in a new era of integration, a practical phase of robotics advancements, where technology providers, OEMs and industry professionals are working together to deliver truly interoperable solutions. In fact, construction has become one of the hottest spaces for advancing both robotics and the move to autonomy. Even small and mid-size contractors are coming to us and asking for solutions.”
He pointed to rebar tying drones and robotic bridge inspection systems as examples of industry acceptance and trust. “It’s no longer a question of what is robotics, but, ‘How do I get it?’ It’s becoming another tool in the toolbox—and the Exosystem is a major advancement that is adding autonomy to jobsites to help build and repair our 21st century world,” he said.
Doosan’s Roecker said his organization will deliver big advancements in 2023 through 2025, through public demonstrations and commercialization. “In a view of autonomous solutions, we’re collaborating with construction partners to commercialize autonomous solutions globally by 2025.”
Topcon’s Hansen concluded: “The tide is turning. Five years ago, construction professionals were still asking about GNSS. That’s not the case anymore. Now contractors of all sizes, from landscapers to earthmovers, are seeing the benefits of automation and digital data, particularly in light of labor shortages, fast-track projects and demand for improved productivity in a low margin business. The next 10-15 years is going to be very interesting!”
Associated Elements provided by physical and digital infrastructure will support advanced UAS ecosystems. New FAA guidance addressing requirements for these elements present a path forward to scalability and highlights associated challenges.
A recent FAA Aviation Safety Memorandum (“Memo”) confirms what ResilienX and other industry experts have long understood; integrating UAS into the national airspace (NAS) is nontrivial, and requires an ecosystem approach. This approach has led to differing views regarding UAS type certification such as which elements of the ecosystem fall within the type certification? For elements which fall outside of the type certification umbrella, but are still within the safety case for complex operations, what are the requirements?
To answer these questions, first consider this guidance from the FAA Memo: A UAS is defined as a UA and its associated elements (AE), including communication links and the components that control the UA, that are required to operate the UAS safely and efficiently in the national airspace system. The AE discussed in this memo are those elements that are not airborne or directly affixed to the aircraft. Where appropriate, the FAA will issue a type certificate for the UA, and the FAA may approve use of the AE through operating limitations and operational approvals (i.e., waivers, exemptions, and/or operating certificates).
While the AE items themselves will be outside the scope of the UA type design, the TC applicant must provide sufficient specifications for any aspect of the AE that could affect airworthiness, including a complete and unambiguous definition of the elements and their interface with the UA, so that their availability or use is readily apparent.
While core AE such as communications links and ground control stations are called out, additional risk-mitigating AE are generally needed to expand operations, particularly for flight beyond visual line-of-sight (BVLOS). For example, no matter how performant a communications link, it will not mitigate weather risks. Therefore, for BVLOS operations, a weather service AE will likely be required. Also, most AE are unlikely to have independent links to the UA and therefore, reliable communications demand consideration for most AE. UAS Traffic Management (UTM) services may also provide AE as means to reduce operational limitations via risk reduction.
The FAA Memo presents a roadmap to scalability, stating that once an AE has been approved as a risk mitigation for one craft/operations combo, it will likely be usable for similar craft/operations.
The AE specifications included as operating limitations in the type certificate and the flight manual would likely be similar for all type-certificated UA and consistent between operating parts[…]
For operations outside of part 107 (e.g., parts 91, 135, or 137), which are more complex, the FAA may provide relief from regulations with conditions and limitations that identify additional specifications to mitigate a possible adverse safety impact.
The FAA may establish additional specifications that are pertinent to the operation of a particular UAS in the operational approval (e.g., exemption or waiver) conditions and limitations, commensurate with the proposed concept of operations and the operating rules from which the FAA is granting relief, such as beyond visual line-of-sight operations or operations which leverage an operating certificate. Certain AE specifications will likely be similar for type certificated UA covered by this memorandum.
The FAA provides a path for UA type certification, and accommodates additional operational envelops via AE to mitigate operational risks. This regime including CONOPS, architecture, and functional/performance requirements are packaged into a safety case for advanced operational waivers such as for BVLOS operations. Once deployed, the FAA obligates the operator to ensure operational compliance with the approved safety case:
The operator is responsible for continued compliance with the operating rules and any conditions and limitations of a waiver or exemption […] Rescission of the exemption or waiver would be the FAA’s primary tool for compliance and enforcement with AE specifications.
Although the operator is responsible for assuring that flight remains within safety case limitations underpinning a waiver, it becomes increasingly difficult as the operator engages 3rd party AE to mitigate various operational risk. Reflecting on the above weather service discussion, most operators are challenged to determine if the subscribed weather service is working correctly. As this ecosystem grows in complexity into the UTM ecosystem envisioned by the FAA and NASA, the operator may also become increasingly responsible for monitoring the health, integrity, and performance of various AE proposed for surveillance, communications, navigation, weather, traffic management, contingency management, etc.
Appendix B of the Memo, “AE Specifications in Operating Limitations and Operational Approvals,” considers some operator requirements for successful compliance. A few of these requirements include:
A requirement that the operator documents and maintains the configuration of the integrated system (such as identifying the UA and associated elements, including communication links and the components that control the unmanned aircraft) that is necessary for each operation to be conducted safely.
A requirement that the operator has an error reporting, evaluation and mitigation program in order to maintain compliance with operating rules and conditions and limitations. The operator must evaluate any failures, anomalies, or other in-service problems to ensure that they do not represent a system deficiency that will result in a subsequent noncompliance with regulations or conditions and limitations. If a failure, anomaly or in-service problem may result in subsequent noncompliance, the operator would be required to correct the issue to prevent that noncompliance and would be required to report the issue and correction to the FAA.
A requirement that the operator documents and maintains a method to assure that each associated element of the UAS continues meet all AE specifications during all flight operations…
When using associated elements that are services provided by a third party (such as a communications link), this assurance may include qualification and in-service monitoring criteria to detect out-of-compliance performance and initiate corrective action (such as avoiding new operations into areas of known coverage gaps until such time as the communication service provider verifies that the gap is eliminated).
The Memo recognizes there will be an ecosystem of systems and services enabling complex BVLOS UAS operations. The FAA delineates where UA type certification stops and AE begin but does not prescribe exact AE means of compliance—instead, it offers areas of consideration for operator and AE providers, as well as examples of the types of requirements.
Based on our experience with the FAA, this approach does not come as a surprise to ResilienX. In fact, for the past three years we have been building a product containing the tools and services to meet these requirements. The ResilienX FRAIHMWORK® (Fault Recovery and Isolation, Health Monitoring frameWORK) provides this capability including the following:
Ecosystem configuration management
Health, integrity, and performance monitoring of the various AE
In-type system-wide safety assurance to mitigate faults, failures, and adverse conditions in-time to make a difference
Digital maintenance tracking and logging
As ResilienX continues to deploy FRAIHMWORK into UTM and AAM ecosystems across the US, we are looking to work with operators looking to realize this FAA guidance and the various AEs which make up the ecosystem and will need to be monitored. Indeed, ResilienX uniquely provides the capabilities and leadership supporting the approval of various AE within the ecosystem and that will need to be monitored. As ResilienX continues to deploy FRAIHMWORK into UTM and Advanced Air Mobility (AAM) ecosystems, we are seeking engagement with operators looking to realize this FAA guidance. — ResilienX is a software company developing safety assurance solutions for autonomous ecosystems. Founded in 2018 in Syracuse, NY, they are focused on ensuring the safe integration of UAS into existing airspace systems and transforming mobility around the world.
Sky Elements and Drone Show Technology Complete Unique Experiment on “The Secret of Skinwalker Ranch”
“Our participation in the latest episode of “The Secret of Skinwalker Ranch” has been an exciting opportunity for us to showcase the potential of Drone Show Technology from SPH Engineering. Through this collaboration, we were able to conduct a unique and innovative experiment using our drones to investigate the enigmatic anomaly areas on the ranch. The use of our cutting-edge software allowed for a safer and more comprehensive scan of the target areas, providing valuable data and insights to aid in the scientific exploration of Skinwalker Ranch. We believe that the successful integration of drone technology in this experiment highlights the immense potential of this technology in scientific research. We are proud to have been a part of this groundbreaking project with “The Secret of Skinwalker Ranch” team,” said Preston Ward, Sky Elements Drone Shows’ General Counsel and Chief Pilot.
