2019 01 in Zipkin
January was a very busy month in Zipkin, in terms of innovation and moving things forward. We had two face-to-face workshops: at Expedia Delhi and at Bas' house in Amersfoort. This led to several works in progress: anomaly detection integration via Haystack, infrastructure setup for formal Apache releases, and near-real time trace processing with VoltDB. Meanwhile, our developing UI, Zipkin Lens, had a facelift and significant improvements in terms of search UX. We'll touch more on these topics below. As always, if this stuff interests you, let us know on gitter or click star if you haven't yet!
Before we dig in, LastMonthInZipkin is a bulletin for projects in and outside our org, in and outside the Apache Software Foundation. Since 2015, we recognize our community is composed of volunteers from different ecosystems. To eliminate any confusion, there is not yet a formal Apache release of Zipkin, and this is not a bulletin exclusive to ASF managed source.
That said, we aim to move everything destined to ASF there by the end of the quarter. If you are impatient for a formal Apache release, please help us finish!
Expedia sponsored many Zipkindred as well ecosystem folks from Homeaway and Hotels.com to a workshop at their office in Delhi. When Zipkin site owner/contributors were asked what they were most interested in seeing.. almost everyone said anomaly detection on trace data.
The first two days, we dove deep into Haystack, with origins in circa 2014 Zipkin. We learned how adaptive alerting fits in today and how it will in the future. Homeaway and Hotels.com told us how Pitchfork let them leverage existing Zipkin tracers with the Haystack backend. A lot of topics related to Kafka. For example, much of Haystack code is KStreams applications. It was also helpful our star Jorge was around, as he could discuss how tracing works in Kafak streaming architectures.
Not everything discussed was technical in nature, we also brainstormed on OSS in general. Expedia and folks built an impressive tracing and analysis system, holding nothing back from open source. We discussed how to attack engagement on this, whether that is using or contributing. By the end of the first two days, we knew integrating Zipkin and Haystack would be a clear win to share valuable code and grow our mutual ecosystems.
The surprise was day three. We didn't plan to code, but we did. By the end of the day, a pilot ran leveraging all four products, Haystack, Zipkin, Adaptive Alerting and Pitchfork. It showcased the same instrumentation 100% sampling for metrics and anomaly detection, yet sampling normally for raw traces. This work will released by Expedia in the upcoming months, so watch for their updates (and star them!). For now, here's a peek:
Our PPMC planned to meet in the Ardennes to regroup, now that we are in the Apache Software Foundation. Weather struck and some were delayed even getting to Europe. We had to change venues as the winter weather made it dangerous to drive. Bas hosted the four of us who could attend. In his kitchen, we progressed Apache deployment setup like Jenkins and various infrastructure we need. Meanwhile, we learned Bas uses VoltDB for timing data at his day job.. timing humans running a marathon. The use case of late reported and mixed quality data is ironically similar between timing data of people and production requests! Towards that end, we spiked a VoltDB experiment for late sampling and near-real time analysis. Follow this thread here until it matures.
Back in Asia, Igarashi and Raja continued to progress our emerging UI, Zipkin Lens. Notably, we changed the UI color scheme to match what users expect, and also refreshed the search UX based on haystack-ui. Finally, Lens now supports site-customizable auto-complete tags, such as what you see below.
Here's a quick round of notable updates, broken down by repo.
- brave-karaf migrated to the ASF org
- zipkin added autocomplete tags api, hardened RabbitMQ and Cassandra logic, lots of work on Lens.
- zipkin-api clarifies trace ID is encoded in big-endian byte order
- zipkin-gcp now sends data to the Stackdriver v2 endpoint instead of the v1 endpoint.
- zipkin-go adds ability to use protobuf encoding for the HTTP and Kafka transports. Default remains JSON.
- zipkin4net is seeking a champion