Raven TestKit is Sentry’s Raven SDK plugin to enable error report interception and inspection of the data being sent to Sentry. The test kit was developed as part of a wider development effort to support TDD (test-driven development) and clean up error and bug reporting. It is the first open source test kit ever written to Raven’s JavaScript SDK, and was fully adopted by Sentry and appears in their official docs.
Most developers agree that using Sentry is great for tracking and monitoring code errors and crashes. Sentry is a system that can easily integrate into your code and control where and when to leave breadcrumbs, and report the error just before everything crashes. But what if you are working in a TDD environment? All that testing is bound to cause a lot of noise in your Sentry logs. And what if you want to log errors offline? Sentry provides only an online solution for reporting errors.
Ziv Levy, doing his thing
How it all started
As a software developer at Wix, I work with Sentry as do many of my peers. As a member of the team now developing Wix’s latest really cool project, Wix Code, I was recently assigned some tasks for which Sentry’s monitoring on its own was just not the right solution for what we really needed.
Wix Code is a new product empowering website creators to develop web applications and insert code into their own sites. They can manipulate the content, make calculations and apply their own business logic while having a complete development environment for client side and server side, including database development. Wix Code developers use Sentry as our tracking and error reporting system to report code errors and crashes.
Every developer knows that writing code during a development task involves a lot of syntax and compiling issues, coding errors, code acting differently than expected, lots of debugging line by line, and more trial and error. We were finding that our Sentry reports were overloaded with these errors that are a natural byproduct of working in a TDD environment.
Add to those already ‘noisy’ Sentry reports, Wix Code, where our users can write their own code that runs within our code, and you’ve got reports that don’t give you a true picture of your errors. If our Wix Code users have an error or bug, it is reported to our system as if the bug originated in our code. At first, while we were running with alpha users, the problem wasn’t that bad. But when we launched our beta version, more and more users joined our system and the amount of errors reported was rising fast. It was impossible to differentiate between our own bugs and user bugs, as well as bugs that were a result of our own developer testing.
The task was to build and add an infrastructure to our application code to somehow ‘mark’ each error when executing a user’s code rather than our own. This would ensure that if the code is going to crash, it won’t be considered our bug and won’t be reported to our regular Sentry system but still have a record of them reported separately. This could also work for testing environments, developer test errors, etc. This way, we can be proactive in distinguishing between real errors in code, development bugs, and our users’ code errors, replacing the need to go through lines and lines of errors in reports that really don’t help us analyze the real bugs in our code.
Any solution we came up with had to:
Be tested carefully to keep the system functioning.
Include regression tests to make sure nothing broke with all of the infrastructure changes.
Most importantly, test the new functionality. Ensure that the right reports were being sent to the right place.
We faced two major challenges.
Challenge 1: How do we simulate a bug in Wix Code?
We couldn’t use a stubbing strategy since stubbing replaces the API method/function with another one. This would result in the original function not running and the error-boundaries solution wrapping the original function. Further, if we would simulate a bug, that would mean that we already know about a bug in the system and would have to fix it! We had to check every API that the user might use to see which other API it operates and then see how we can interrupt the normal code execution flow.
This included:
Stubbing internal third-party libraries that we use.
Building proxies and facades in our code so we can surgically point out user code execution.
Damaging the app state in memory.
Challenge 2: How can I test the report data?
We had to somehow access the data being sent by Raven, Sentry’s SDK, to the Sentry repository. We knew we had to find a test kit out there which would intercept the sent items and inspect them in such a way that we could confirm that the content of the reports were our own bugs that needed fixing. In our case, a report is a bug or any other crash, and it would consist of the error information, the stack trace, events and bread crumbs along the way up to the crash. We also needed extra data that we manually add to the report to help us as developers understand what the user’s state was at the moment of error.
So after a lot of research, we found that Sentry doesn’t have their own test kit for the Raven SDK. Also, we couldn't find anything like it that was available elsewhere on the web. We concluded that the best solution was to get our hands dirty and create it ourselves! After learning and understanding how Raven works, or at least as far as we needed to, we had to map out our strategy for how we were going to hop on Raven’s reporting lifecycle and start coding.
The Solution: Raven Test Kit
At first, we weren’t sure what the right solution; Is there a “best solution”?
We considered a few options:
Write a dedicated server so all the reports would be sent there. The advantages of this option are that we don’t need to “get to know” Raven and the way it operates and sends its reports. It would also be the most close-to-production condition of our system. The major disadvantage is that to run tests we would need to start a server each time, even for non-browser tests, requiring development time and resources.
Use a dedicated Sentry repository for testing. Seems like an intuitive solution, almost zero development work. But then to verify tests, a developer would need to launch a browser, go to a Sentry interface, and look up the relevant reports. Well that option doesn’t really scale when several developers would have to perform each of those steps simultaneously and let’s not forget - we still have to be online for this option.
Change the way Raven acts upon exception. We started to look at the source code to find interesting integration points. l We discovered this built-in capability of setTransport. The advantages are clear, we can use Raven’s built-in capabilities, no hacks. We don’t need any further setup for our tests, and no dedicated servers. Also development time for this solution becomes much shorter (yay!).
At first, we didn’t think about creating a third-party module and just wrote a simple, utility module in our own code base. After speaking about it with more and more of our colleagues, we understood that there was a real need for a module to be used widely in many other testing scenarios. This is true especially here at Wix Engineering where even the smallest bit of code is thoroughly tested.
We opened a new public open source repository at GitHub, Raven-testkit, and completed all of the work on the library API there. Take a look for yourselves. The API is simple and easy to understand and use.
Raven Test Kit enables Raven to work natively in your application by overriding the transport layer using Raven’s own API:
It overrides the default Raven transport mechanism. In our case, our goal wasn’t to send the report. So we constructed a local, unreachable variable in this singleton instance:
Then upon our instantiation of the Raven instance, we save the report locally, and if someone is waiting for “onSuccess”, then we call the “onSuccess” callback:
Eventually, the report is not actually sent to Sentry, but rather logged locally. In this way, the logged reports can be fetched later by any one of the Raven Testkit APIs, such as:
Once you have the report object, you can verify the properties being sent or use it some other way that benefits your testing environment.
Ironically, this testing kit library consists of its own tests, and as part of its Continuous Integration and Delivery (CI/CD), it is tested for functionality upon every change.
While using Raven-testkit ourselves, we keep evolving and adding more APIs. We believe that for now, it is powerful enough to use in production. But additional syntax-sugars are planned to be added, per usage feedback and engineering needs.
Sentry’s Tips & Tricks
After quite some work on our end, on API documentation, code reviews, code improvements, together with Sentry developers of the Raven’s JavaScript SDK, another milestone was achieved for Raven-testkit - it was published in Sentry’s official Tips & Tricks page as the best-practice to run tests with Raven! We are very proud of this achievement and were very happy to work together with Sentry’s teams. A special thanks goes to Kamil Ogórek (@kamilogorek) for helping to standarize the proper documentation.
Benefits
Any development projects using Sentry can really benefit from the test kit. One of its features is that it provides the ability to work offline with Raven without having to hack into Sentry and Raven. You can run integration tests without exploding Sentry with false reports by running negative flows in your tests. Since it was first published, it is already in common use in several development groups in Wix’s R&D, and new applications and approaches have already been adopted for Raven-testkit.
As time went by, we discovered more and more uses for the tool by other teams for their own needs. For example, you can connect Raven-testkit and run your system offline while working with Raven, as Raven-testkit intercepts sending the report to the real Sentry system. Up until now, to run your system with Raven, you had to wreck Raven configuration so it could run with tests. Otherwise, running tests with a working Raven could result in having all of those reports coming in from tests to be sent to Sentry. Using Raven-testkit eliminates the need to wreck Raven’s configuration and intercept reports before being sent to Sentry.
This was a tremendous opportunity to write something pretty cool that may help all Sentry users to better test their apps, without the need to hack Raven. Raven-testkit is still the best open-source library out there that enables development teams to inspect their reports with the ability to determine the report format and content as the code evolves.
It’s really amazing when people come and ask about your own creation, looking for new approaches on how to use it and sometimes even suggesting new features and discussing use cases and capabilities. So give it a try, send your feedback and let us know how you’re using it!
This post was written by Ziv Levy
You can follow him on Twitter
For more engineering updates and insights:
Visit us on GitHub
Subscribe to our YouTube channel