Open Raven Platform Release: It’s all about scaling — core scanning engine, map, data catalog, across the UI, discovery, policies, and rules

This month’s theme is all about scaling — in the core scanning engine, map, data catalog, and across the UI in general. Beyond making the product faster, easier, and all-around better for large environments, we are also wrapping up the journey to make our main open-source project Magpie, a pipeline into the platform for new discovery features, rules, and policies.  

Data Scanning

Incremental scanning

The close of April brings differential scanning into the platform as a complement to the exhaustive baseline scans we have done to date. Scans will now check if each target object has been scanned using the data collection in the past and skip over that object if so. If the object has been updated, then it will be rescanned. Depending on the objects being scanned, this can immensely reduce the duration of the analysis as well as the associated Lambda costs.

Sampling and scan limits

Sampling a percentage of a bucket now reduces the amount of data and objects checked against scan limits such that even extremely large buckets can now be analyzed. Coupled with incremental scanning, this allows for eventual, full analysis of any sized bucket over time.

Reliability and performance

Over the course of this month, we’ve made a bunch of minor “under the hood” improvements to accelerate scans and boost their reliability.

Map

Visualizing thousands of cloud assets can be powerful — or overwhelming. With this release, we’ve improved the logic for how large environments are laid out within the Map for easier viewing. The default zoom level limits asset display, the assets are now much closer, and lines between them are as short as possible.

‍

Open Source: Discovery, policies and rules 

Our main open source project Magpie is now a functional pipeline for new discovery capabilities, policies, and rules. The benefits are most obvious for our rules and policies, which over this week and next will expand dramatically to include what’s visible in GitHub now here. On a quick glance, you’ll see many rules covering important data store configuration problems, as well as an early look at some Google Cloud Platform (GCP) rules. The GCP discovery and rule capabilities shown within the repo aren’t making the transition to the platform. Yet.

Bugs / Enhancements / Tweaks

• Sorting a column now defaults to descending first instead of ascending
• Scanner settings are now found in the Settings area of the UI, along with everything else
• Scheduled policies will now correctly start at their specified time instead of starting at the previously scheduled time
• Toggle switches in the product now turn to green when enabled instead of black
• A slight change in colors throughout the product. For example, blue to dark blue.
• The asset list now prioritizes the object name when rendering columns instead of partially hiding them
• Fixed a bug that removed violation icons from not showing in the asset list
• Tag tooltips in the asset list have been re-enabled
• Tooltips with many values no longer run below the page
• Tooltips size on the Data Scan Job page are now consistent
• The default value for the DMAP Scan Queue setting now correctly states “Select”
• Pending scans now correctly have the options settings available
• The Ignored Violations tab in Policy Violations no longer cuts off text early
• Data Catalog values now take advantage of the entire screen space
• Data Catalog export now includes the Account ID field
• Filtering on data classes that have no results now correctly filters out all results