Open Raven Platform 0.9 and a Few Release Updates

Open Raven Professional 1.0 is another week closer to general availability and yet again more functionality has been added into this week's preview, 0.9.

This week it's mainly under the hood stuff but we're still on track to be “mainly code complete” ™ on November 3rd with GA on November 17th. “Mainly code complete” ™ is not my attempt at an alternative truth joke but a wink-wink acknowledgement that some things will still be in-flight right up until GA…like Webhooks for Slack and PagerDuty integration for instance…things we never originally planned to do but think are important and users deserve so we are doing them anyways. We are just like that.

As always just go to your clusterand turn on the ProPreview feature flag, and as if by magic, your UI will change in front of your eyes and data classification features will appear. It's like David Copperfield but without the hairy chest.

As well as a metric ton of bug fixes and performance improvements (see below) this release is really about adding some new tightly QA’d data classes to find developer credentials :

• AWS secret keys
• ODBC connection strings
• JDBC connection strings
• OpenSSH keys

For open SSH keys, we have unique data classes developed for various formats like PPK, the same techniques we are applying for X.509 certs (i.e. expired certs, types of certs etc). There are 50 data classes across privacy data, financial data, health data and developer credentials in development right now and of course you can write your own. Adding your own basic regex ones are as simple as, well, writing the regex so go wild.

We have (in R&D) a validator function as the first demo of the ability to actually test data that has been found is real data. The AWS validator will take an AWS secret keys found and attempt to login to your AWS, validating its real and determining the account the key is associated with. What, you enjoyed spending your days trying to figure out if credentials were real and what they were for? Of course not…

I mentioned last week we're working on a data fabrication tool that we plan to open source. It's been a good week on that and moved forward. You can inject data sets into documents of various sizes including inside the formats such as MSFT Word tables and charts so we and you know exactly what we can find. It will get extended for every scenario we and you can think of, with the goal to be totally transparent and allowing us and you to generate test data that you can test both our accuracy and compare us against other tools like Amazon Macie. To be clear, we plan to eat Macies lunch and are not afraid to talk about it. Joking (not joking) aside, the truth is we know the problem we're tacking is a hard one and we won't get it all right out the gate, but by being open and transparent and as fast as hell, you can judge our performance out of the gate and predict what the best long term solution will be.

The next release will look the most complete to date with almost all UI features we are promising for the Platform 1.0 release so get ready. And yeah, you see AWS Marketplace in the change log. More on that next week.

And if you're interested in bugs, nits and minutia (aka engineering reality), we've been working on them as well. See below, but just know we've got it covered, so get back to looking at the very pretty maps!

KNOWN ISSUES:

• When the window size is too large some columns lose alignment in list view
• Dynamic asset-groups do not currently save.

There are a bunch of bug fixes including:

• Making the DMAP scheduling toggle work properly

Still behind feature flags:

• Lots of updates to Data Class including lots of new classes
• Improvements to the way we handle scanning
• Movement on policy UI

Under the hood:

• Lots of improvements to enable SaaS
• Updates to the Flatcar AMI to address CVEs
• Update to Kube 1.19 for new clusters

#### Raw List of Changes ####

‍

# aws-discovery-svc

• [!129] Add AWS SecretManager discovery
• [!133] Add AWS storagegateway discovery

# cross-account

• [!35] Of/fix concurrent account polling

# dmap

• [!37] Of/configurable batch size
• [!38] Of/scheduling

# dmap-scheduler

• [!30] Latest S3 Jar (404 handling, regex fix)

‍# productui

• [!834] style CodeEditor and use AceEditor as implementation
• [ENG-4444] (Sub-task) Validation for required fields should not use the Red Error pattern
• [!838] initial policy list integration with clone, edit, delete, create, and sort
• [!839] initial rules integration with sorting, delete, edit, create
• [!840] Reload table after saving new data scanning item
• [!841] scaffolding for Triage
• [!842] AWS Discovery should show all accounts from cross-accounts API and ES
• [!843] add validation for the policy and rules forms and adjust their mappings to schema changes
• [!844] Policy pages tweaks
• [ENG-4506] (Task) Create Pro Edition v1.0 feature flag
• [!846] Feature/ENG-4488 3D-UI Design Updates
• [!848] Epic/maps v2
• [!785] update README.md file instructions to run product UI locally to account for...
• [!849] Triage Section

# helmfiles

• [!209] Add policy service helmfile boilerplate and add utility script to help with boilerplate
• [ENG-4442] (Story) Write violation reports and audit log entries
• [!212] spread out the cluster_type usage
• [!213] turn down logging level on OPA and add /api/policy-rule to policy ingress
• [!214] Stop the noise

# cfn-pivot

• [ENG-4286] (Initiative ) Clusters provisioned this way should have a `cluster_type:saas` in Datadog
• [ENG-4297] (Initiative ) AC: Stop publishing the `admin.conf` to the cloud formation outputs
• [!36] Fix some loose ends from !35
• [!37] Update kubernetes to v1.19.3
• [!38] Update OIDC to reflect SaaS environment
• [!39] Make DmapStack optional and fix a sentry kaboom

# aws-marketplace

• [!55] Changes required to support cluster_type=saas
• [ENG-4302] (Initiative ) Product Changes: Update clusters to 1.19
• [!57] Initial bare-bones SaaS stack
• [!58] Apply ACM by accountId
• [!59] Flatcar 2605 7 0
• [!60] Update saas with new AMIs, fix patch process

# s3-scan-service

• [ENG-4457] (Sub-task) Classifier Loading [S3 Scan Service]
• [!24] ENG-4483 : Clean up gitlab-ci.yml for s3-scan-service
• [!25] prevent sort errors on indexes that haven't been mapped
• [!26] Removed possibly troublesome DataClasses and added more credentials
• [!27] Of/fix idempotence check
• [!28] Of/fix object total size