We met as undergrads at Georgia Tech and come from a DevOps and operations background so we’ve seen this first hand. Each of us has over 15 years of experience building high-reliability systems, starting in the early days with satellite earth station monitoring. As interns we once wrote a bug that caused a 32 meter antenna to try to point down through the earth, almost flattening the building we were in. It was a great environment to learn about engineering reliability. We leveraged this experience to tackle monitoring Java app servers, SOA, SaaS observability and cloud data warehouses. What if we could use a form of observability data to automatically test the reliability of new deployments before they hit production? That’s the idea that got us started on Speedscale.
Most test automation tools record browser interactions or use AI to generate a set of UI tests. Speedscale works differently in that it captures API calls at the source using a Kubernetes sidecar [3] or a reverse proxy. We can see all the traffic going in and out of each service, not just the UI. We feed the traffic through an analyzer process that detects calls to external services and emulates a realistic request and response -- even authentication systems like OAUTH =). Unlike guessing how users call your service, Speedscale automation reflects reality because we collected data from your live system. We call each interaction model a Scenario and Speedscale generates them without human effort leading to an easily maintained full-coverage CI test suite.
Scenarios can run on demand or in your build pipeline because Speedscale inserts your container into an ephemeral environment where we stress it with different performance, regression, and chaos scenarios. If it breaks, you can decide the alerting threshold. Speedscale is especially effective in ensuring compliance with subtle Service Level Objective (SLO) conditions like performance regression [4].
We're not public yet but would be happy to give you a demo if you contact us at hello@speedscale.com. Also, we are doing alpha customer deployments to refine our feature set and protocol support - if you have this problem or have tried to solve it in the past we would love to get your feedback. Eventually we’ll end up selling the service via a subscription model but the details are still TBD. For the moment we’re mainly focused on making the product more useful and collecting feedback. Thanks!
[1] https://services.google.com/fh/files/misc/state-of-devops-20...
[2] https://aws.amazon.com/builders-library/automating-safe-hand...
[3] https://kubernetes.io/blog/2015/06/the-distributed-system-to...
[4] https://landing.google.com/sre/sre-book/chapters/service-lev...
- Request 1 is a post that generates a "todo" with a random id "5y22"
- Request 2 is a "get" for /todo/5y22.
That works in production, but on replay of the traffic:
- Request 1 generates a different random id, "86jj".
- Request 2 is still a replayed "get" for /todo/5y22, which is now 404.
How does your tooling handle this nondeterminism in replays?
So if you mean that we want to "test" TODO, our analyzer looks for data in subsequent requests that was provided by the system-under-test (SUT) in previous responses. A common example of this is an HTTP cookie. The SUT gives us a session id through the Set-Cookie header response. So in a subsequent request we use the cookie from the app, not the one that was recorded. This has been done in general way to look for tokens.
Of course nobody is perfect so we'd love to see your real world app and test our algorithms against it. :)
Non-determinism would be, for example, something that's time-sensitive. If some result varies by time, then the only way to test it is to include time as a parameter. This can be complicated if the time variable plays into asynchronous updates (e.g. you want to test that a POST update worked, but it's actually eventually consistent). Caching (e.g. through Varnish or a CDN) would be another thing to make such tests much more complicated.
Another example is an API that has side effects. For example, a stock trading API might read real time quotes from another service. A stock trade then alters the next quote.
Thanks for sharing the info about your project, I am checking it out on GitHub right now. :)
Ken worked as a top SE at New Relic and has been involved in the Atlanta Web Performance meetup for many years. It was a wonderful surprise to see you all on HN
Anyways, was trying to email you to founders at your domain but no response. Can you reply to my email? Handle is lli at my company domain (also in my profile). Would love to have a quick chat this week to get well warmed up before thee Day.
One benefit of the artificial environment is efficiency (can cover e.g. a week of historical data); this also requires mocking out time with simulation/replay time. The integration with data-platforms makes a big difference as it means data-scientists can help set up the right test scenarios.
So how open is that alpha? I’d really love to try this out.
Please send a note to hello@speedscale.com so we can get the details of your environment and determine if it is a fit for our alpha.
1. https://github.com/buger/goreplay
One of the early observations from my co-founder Nate was there are 3 key ingredients to testing in a SOA environment [1]
* Automation
* Dependencies
* Data
While goreplay has a form of automation, it doesn't help you with dependencies (no mocking), and the data is either streamed or locked in a special file format. Of course like any open source project there's pieces of the solution but you have to assemble them. For instance there is no UI, no reports, no pass/fail assertions, no integration with CI system, etc. I'm by no means an expert, there is likely a path to combine with other open source projects to fulfill those.
Our stuff isn't perfect, but we primarily see overlap with the automation capability of Speedscale and the goreplay project.
[1] https://speedscale.com/2020/02/06/triplethreat/
(edited formatting)
Wondering how do you deal with stateful services? You mention an "analyzer process" for external services, what about internal services?
It seems this would work well at a single service level, but would it also be possible to apply the analysis at both a single service and a group of services? Some form of unit-testing and integration-testing at a service level...
We've been down the path of stateful services before and actually reflect the proper state in our responder. Because we control the test that is being played against the system-under-test, we understand the sequence and order of calls that will be made to the downstream system as well.
In addition, the analysis actually captures all outbound services at once. We are able to identify each separate hostname that is invoked and mock them all out as a group. One of our first alphas was stunned that it auto mocked 7 backend systems on the first try.
"Reach new heights with weightless test automation. ProdPerfect is the first autonomous end-to-end (E2E) regression testing solution that continuously identifies, creates, maintains, and evolves E2E test suites via data-driven, machine-led analysis of live user traffic."
Do you have more background on the multi geo use case?
Containers with env vars are easy though!