This week at AWS Summit in New York, we got a fascinating insight into how Honeycomb.io helps engineers debug their systems through the use of Big Data.

Honeycomb’s service is “differentiated by its scale and speed,” explained Liz FongJonesleading advocate for Honeycomb Developers, during the Summit’s keynote.

The goal of the Honeycomb service is to have any engineer answer any question about their malfunctioning or under-functional system within 10 seconds or less – even previously unasked questions that arise from iterating a train of thought, or, for “Following the Ariadne’s Thread”, as explained in his keynote.

The secret sauce? The o11y company collects all possible operational data from the customer, stores it on AWS SSDs, and then uses a combination of the AWS Lambda Serverless Serviceand fast AWS Graviton ARM-based processors to parse data and return queries.

The Honeycomb service relies on a variety of AWS pre-packaged analytics services. Some of them are already captured on internal AWS services, including the Relational database service and CloudWatch.

But also instrumental is the Amazon Distribution of OpenTelemetrythe Cloud Native Computing FoundationThe open source package of APIs, libraries and agents for monitoring applications via traces and distributed metrics.

Honeycomb pre-processes all data generated by the app and stores it on Amazon Simple storage service (S3), where it is then analyzed on the fly via the AWS Lambda serverless service. The service currently processes 2.5 million traces per second, up from 200,000 just three years ago. “Our customers are asking 10x more questions about 10x more data,” Fong-Jones said.

That’s a pretty impressive setup for the work of just 50 engineers. The configuration consists of a combination of stateful and stateless services, built primarily in GoLang, but also with Java and Node.js.

For stateless services, Honeycomb uses the Amazon Elastic Kubernetes service running on both EC2 C6g Graviton2 and C7g Graviton3 instances.

Honeycomb seems to be bullish on the ARM architecture.

Fong-Jones noted that when the company saw a 10% improvement in median latency, when switching to Graviton 2 from the AWS M5 Intel Xeon-based instances. “The Graviton 2 processor is just a lot more efficient and we’re able to push a lot more load,” she said.

Additionally, A/B testing between Graviton2 and Graviton3 revealed an additional 10%-20% improvement in tail latency, and a 30% improvement in our throughput and median latency. And CPU usage is about 30% lower, “meaning we can push it a lot harder,” she said.

Honeycomb saves some money by using AWS location instances, which are machines not already in use in AWS. AWS has a graceful shutdown handler that exits workloads when CPUs are needed elsewhere. Here, Honeycomb initially saved around 20% by moving some workloads on-premises.

For Kafka streaming data ingestion, Honeycomb uses EC2 Im4g instances, which are based in Nitro SSDs. Earlier, slower storage iterations left the CPU hungry for work. “Sizing everything correctly on Im4g allows us to properly hit our network’s CPU and storage thresholds,” she said.

Lambda provides another piece of the puzzle. Even using 100 fast Graviton instances won’t be enough to get the job done, given the millions of files stored on S3. That’s where Lambda comes in, being able to instantly provision up to “tens of thousands of parallel workers.”

“With AWS Lambda and Graviton combined, we see about a 40% improvement in pricing performance,” she said.

As someone new to the world of observability, I have to say that traces mean a lot more to me than all the different ways of generating and aggregating metrics.

— Philip Carter (@_cartermp) July 14, 2022

This week in programming

• A Cloud ARM race?: For about a decade now, the industry has more or less agreed that ARM64 single-threaded multi-core processors in the data center would be a good thing, given their operational efficiency. AWS offered ARM in 2018, and with the introduction of Graviton2 in 2020, AWS indicated that the ARM architecture would not be suitable for mission-critical scalable cloud workloads. In April, ARM also came to Azure, with the general goal Memory-optimized Dpsv5 and Epsv5 virtual machines. Now the last of the big three cloud providers, Google Cloud, also joined the party. This week the company introduced its first ARM-based instance, the Tau T2A. Powered by Ampere Altra ARM-based processors, T2A virtual machines offer up to 48 vCPUs per VM, with 4 GB of memory per vCPU and 32 Gbps network bandwidth. The Tau family T2A virtual machines are suitable for scale-out workloads such as web servers, containerized microservices, data logging processing, media transcoding and Java applications, according to the company.

omg tad.

ARM is inevitable. It’s the future. It’s now on all major cloud providers – Amazon, Microsoft, Google and Oracle.

It’s in your laptop if you’re using a modern Mac.

Welcome to the future, everyone. https://t.co/tjywueZLuB

— Liz Fong-Jones (方禮真) (@lizthegrey) July 13, 2022

• Visual Studio gets comfortable with Git: Microsoft makes its VisualStudio integrated development environment more compatible with the widely used open source source control management software git. Visual Studio 2022 17.3 release. Until now, switching between git branches in a repository caused latency as the new branch would be loaded. the team would most likely have experienced a solution reload when moving to or from this branch,” Microsoft Senior Program Manager wrote. Gherfal Taysserin a blog post explaining the output. Microsoft figured out how to reduce the number of reloads by 80%, eliminating inefficiencies like requiring a reload every time a team member loaded into that same branch. This version includes some improved performance for indexing and colorization C and C++ code as well. For example, in an earlier version of Visual Studio, it would take 26 minutes to index the Chromium codebase, while the newer version can complete the same task in six minutes. reports SD time.

• Observability, the ultimate frontier: In the suburbs or in the city, you might not think so much of the night sky, which appears as a vast expanse of darkness only occasionally punctuated by a star, or one of Elon Musk’s Starlink low-earth satellites. Without the surrounding urban light pollution, however, the night sky has an entirely different and far more ominous view. Rather, it’s a chilling whirlwind of planets, stars, galaxies, and various other bits of matter and energy all whirling through space and time. Frankly, it’s a miracle that our little planet has survived in this maelstrom so far without being at some point for a hyperspace bypass (as predicted by Douglas Adams in his quintessential “Hitchhiker’s Guide to the Galaxy”). This week we got the deepest glimpse of that cosmos, thanks to NASA’s James Webb Space Telescope (JWST), which delivered its first batch of infrared images of the universe (or a very small part of the universe) JWST’s lens, orbiting the sun about 1 million miles from Earth, is so powerful it actually saw in time. The initial images she took, of galaxy cluster SMACS 0723, showed the galaxy as it looked 4.6 million years ago, thanks to limits on the speed at which light can travel ( 186,000 miles/second). In May, our European correspondent Jennifer Riggins filed a fascinating report on the engineering behind JWST. “It’s a great platform to demonstrate site reliability engineering concepts because it’s about extreme reliability,” IBM SRE Architect Robert Barron said of JWST to WTF. is the SRE conference. “I think there are a lot of lessons and a lot of inspiration that we can take from this work in our day-to-day lives.” For example, he explained, the design team first focused on functional requirements: mirror larger than the Hubble Space Telescope, but NASA did not have the capability to send a mirror so big in space. This led them to define some non-functional requirements, such as creating the mirror from smaller hexagons that could then be unfolded in place. The work was based on NASA’s values ​​for sending spacecraft into space, namely that components should be redundant, reliable and repairable. “There is no doubt that the James Web Space Telescope’s SRE strategy has more at stake than any strategy adopted on Earth. This still serves as a fantastic example of how site reliability engineering and observability needs vary in the context of circumstances,” Riggins wrote.

The New Stack is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Honeycomb.io.