48% of organizations analyze data streams in real time

Organizations are increasingly opting for an analyze-then-store method for streaming data, according to a new survey from Swim.
Swim, the developer of a platform for continuous intelligence apps, found that almost half of all respondents to its quarterly streaming data survey analyze data in real time, instead of storing it first.
This is an improvement on previous surveys and shows that the industry is moving towards a method of analyzing and then storing, to avoid data obsolescence. The survey found that 16% of organizations perform contextual streaming analysis, further reducing the time between streaming data into the system and analysis.
SEE ALSO: Where is streaming data analysis most tangible?
Some data does not need to be analyzed in real time. 32% of organizations store data in a warehouse or lake and process it later in batches, to gain contextual insights.
Continuous processing and streaming of data has many key advantages. According to the organizations surveyed, the most important benefit is an improved customer experience (44%), followed by early warning of failures or threats (36%) and a more efficient supply chain. (36%).
Most use streaming technology for more than four purposes. The main ones are data pipelines (54%), messaging (54%) and microservices (52%). A third of responding organizations said they use it for streaming applications.
21% of organizations process more than a million messages per second, a staggering amount to process and analyze. Two percent of organizations process more than 10 million messages per second.
70% of organizations create their own custom streaming data solutions. 50% build the full stack with available open source infrastructure and 20% run on a pre-integrated commercial platform. The remaining 30% subcontract to commercial platforms for all stages.
Apache Kafka is by far the most popular open source streaming technology, with 83% of organizations using it. Apache Pulsar takes second place, with 17% usage. Apache Cassandra, Apache ActiveMQ, and Apache Spark Streaming were also cited as popular frameworks.
In terms of skills, people who are technically proficient in Python and Java are rated highest, followed by Kafka Streams and JavaScript. Scala, Typescript, Golang, and Rust all had less than 15% interest from organizations, confirming how the industry has turned to these two popular programming languages. This makes sense because the hardware, software, and SaaS industry actively processes and analyzes streaming data at a much higher rate than other industries.
Hardware, software and SaaS companies are leading the way in streaming data processing and analysis, accounting for 20% of respondents. Retail/e-commerce and finance were followed in second place in terms of adoption.
As the shift to edge computing continues, we expect increased adoption of analytical streaming capabilities and real-time or at-source analytics effort. With this, organizations will have a continuous feedback loop, instead of weekly or monthly batch updates, leading to faster and more accurate decisions.