Break down data silos with an Apache Arrow platform
Voltron Data officially launched last month with a mission to make Apache Arrow easier to use for big data analytics projects, and in particular to focus on improving interoperability with other systems. The startup, which has raised a total of $110 million in funding, spoke with The next platform on how it aims to change the way people interact with data.
Apache Arrow was introduced in 2016 as a framework for developing high-performance data analysis applications based on a columnar in-memory layer. Since then, a growing team of Arrow developers have transformed the project into a toolkit of modular libraries that provide a foundation for advanced analytical processing.
Voltron Data claims to employ the largest group of Apache Arrow contributors in the industry, and just unveiled a new business subscription service for companies developing solutions with Apache Arrow. This offers what Voltron calls a targeted set of services designed to speed up time to success with Arrow, divided into three editions, including a Free Edition, a Dev Edition, and a Pro Edition.
Josh Patterson, co-founder and CEO, tells The next platform that many people simply don’t understand the scope and complexity of the Apache Arrow ecosystem.
Voltron Data is also aiming beyond the obvious customers for high-performance analytics, such as financial services, and sees a broader role for tools like Apache Arrow in solving the kind of data problems that most organizations will have.
“It’s natural to see the financial services industry as a key player in fast and efficient analytics. But I think it’s just a more general thing we should be doing, because it’s hard to find a Fortune 500 company that doesn’t have a big data problem,” Patterson says.
“When you start thinking about things like cybersecurity, the explosion of IoT data, log data as a whole is just one of the fastest growing types of data, and we need to analyze it. – we can’t just say, well, this is too much data, or I can’t do this, or I can’t do that. And at some point, you need order-of-magnitude gains to deal with this increase of an order of magnitude in the data that people collect,” he added.
The solution will be to make systems more efficient and to be able to operate new hardware efficiently, according to Voltron Data. The company also aims to simplify the movement of data between different application silos, and thus bring benefits to all.
“I think it’s an exciting time in general, for data analytics as a whole, because we have things like Rapids and other innovations on the way. What if we could also reduce these barriers? What if we could lower the barriers to allow new innovations to come to market and better combine these systems? We are bridging the HPC world to the data analytics world, where machine learning and AI are going. I think it’s really exciting to see how we can just build better systems that deliver more value,” Patterson said.
Voltron Data Co-Founder and Chief Commercial Officer Darren Haas went further and said Apache Arrow would enable data collaboration between different business units or divisions, even if those units had standardized on different tools and languages. .
“So GE had all these different business units. And they all wrote in different programming languages. They wrote in Java, they wrote in R, they wrote in Python. If you step back and look at Arrow, it’s not just like bringing the languages together, it’s actually bringing the business silos together. If you embrace Arrow and the ecosystem, your team here can now talk to another team here as data,” Haas said.
According to Voltron, organizations subscribing to one of its enterprise subscriptions will be able to report and track issues through a customer portal, and a team of engineers will work to resolve any issues.
As with other developers offering commercial support for open source projects, Voltron offers to backport bug fixes into the latest major Arrow release and ship stable hot-fixed versions packaged and delivered as required.