Breaking Free from Vulnerability Scanning Noise: Automated VEX Aggregation for Accuracy - T. Fukuda¶

Playlists: KubeCon + CloudNativeCon NA 2024 - Salt Lake City

Presents a centralized VEX repository and automated aggregation to cut vulnerability scanning noise.

Categories: Security

Topics: vex, trivy, vulnerability scanning, sbom, vex hub

Transcript¶

▶ 0:01 all right so uh hello everyone uh thanks for coming to my talk uh today I'll be talking about bs and Bs have a centralized repository that aims to make BS more accessible and practical for everyone so here's the agenda for today and I provide an overview of backs first then discuss civil challenges we face with its adoption and finally I present our solution to the these challenges so let me introduce myself I'm T CER working at a security as an open source engineer I started to Tri in my spare time H it's now open source security scanner mainly for

▶ 0:45 vulnerabilities and the misconfigurations and now I Lu maintain Tri as part of my job so I would be really happy if any of you have ever heard of try so uh let's get start about it so uh over the past decade we've seen 4 60% increase in the number of vulnerabilities cbes published annary and L for share bomb supply chain blah blah blah so yeah it's enough so I think I'm sure many of you are already familiar with this story and Trend so I don't want to waste your time repeating that so what Ms here is we need a

▶ 1:29 effective and efficient vulnerability management so it involves uh multiple steps so let me walk you through these steps to understand how the workflow is going so it starts uh with a lot of vulnerabilities as I mentioned like 12,000 or 30,000 vulnerabilities annually H but here's a good news you don't have to look at all of them so the first step is uh detecting which vulnerabilities are actually affecting your systems or assets so we can use S bombs uh SCA tools or security scanners to help with this

▶ 2:13 process and the next comes the initial risk assessment H it can be automated by using cbss metrics like a cbss scores vectors and cilities and the new metrics like epss scores uh ideally we should fix all vulnerabilities immediately but it's not practical so we do prioritize we filter out negligable vulnerabilities then fix them later so the last step is uh uh threat and asset based risk assessment while Frameworks like ssbc can help uh this step often requires

▶ 2:56 human judgment we have to answer questions like is is this system internet facing uh does this system have access to personal information how does this vulnerability affect our systems and can we even accept this risk so it's challenging to automate so it means it takes more time than the previous steps uh as we and also high risk vulnerabilities that remain at the end requires emergency fixes like out of schedule fixes which which means even more time spent uh on like uh testing and a deployment so as we move to the right on this workflow each step takes

▶ 3:39 more time and resources so reducing the number of vulnerabilities that reach these later stages a significantly decrease over all workload of the bability management I'm just seeing shifting left in this worklow is essential so I mentioned using uh sbor and security scanners their detect vulnerabilities affecting our systems but here's a important caveat so detected vulnerabilities might not actually be exploitable I can share some common scenarios first uh there are cases where cbid was assigned but it turned out to be incorrect and wasn't actually

▶ 4:24 vulnerability the second there are vulnerabilities that only affect systems and specific specific configurations the third like we depend on the bable package but the bable code is never called in our application so it means exploitable now detected viabilities does not always equal exploitable pabilities H here's a real world example bability uh was discos found in npm package and also CB ID was assigned but it after the maintena looked into it it was later determined to be incorrect and the security advisory also has been

▶ 5:07 revoked but this vulnerability Still Remains in the National vulnerability database NBD and some other advis databases so which means the security scanners still might frag it as a security issue and here's another example from kubernetes so kubernetes was using the vulnerable version of do distribution which is a go mod go package library but the vable code was not to include it in kubernetes so it means it's not uh exploitable attackable even though security scanners would detect this issue so research said only 3% of Open

▶ 5:51 Source vulnerabilities are actually attackable and exploitable also this report mentioned while many many systems were using the vulnerable version of log 4J 96% of them were actually not affected by log for share also another research shows that many packages installed in container images uh are not loaded at run time at not used at run time so this tells us something important just depending on a vulnerable dependency vulnerable package doesn't necessarily mean your system is bable so

▶ 6:33 but uh please don't blame s bombs and SCA tools uh they've improved V vulnerability visibility for sure and now we are simply ready to move to the next stage so let me explain reachability with a simple example H let's take a look at two software products H in the first case the software a depends on component C which calls the vable function E1 and the component e in this case yes it's actually bable because there's a clear path to the vulnerable code so now let's take a look at software B in this case the software B depends on

▶ 7:18 component D but this component D never calls the vulnerable function E1 it calls the function E1 in the component e so in this scenario if even though software B is using the same bable component e it's not exploitable yeah this shows us that the bability reachability is not just about uh what components you use it's about how we use the components so the relationship between the parent components and their dependencies determines whether affability is indeed exploitable reachable so let me go back to the diagram I

▶ 8:02 showed earlier if we can evaluate exploitability before assessing vulnerabilities deeply we can significantly reduce the number of vulnerabilities that need to be assessed in the subsequent steps so as you can see we have much less input flowing into the time consuming steps that assess variabilities so ideally we want to reduce 97% vulnerabilities like in the research report but even a small smaller reduction in noise can make a great impact on the bability management

▶ 8:45 efficiency so how do we express the exploitability information and that's exactly where beex comes in uh beex is originally from ntia and uh it can uh describe whether a product is actually affected by a specific vulnerability uh as we saw in the kubernetes example uh software maintenance might have analyzed source code and evaluated reachability however that information often exists only on GitHub issues or documents or block post so it's hard to automate but since beex provides this information

▶ 9:30 in a machine readable format like Json we can automatically consume that in our workflows so beex document uh consists of multiple be statements and the be statement is composed of three key elements bability product and Status so beex is idea like a concept so there are four implementation as far as I know so cesa as PDX cyon DX and open BS the wi open BS is a specifically designed for BS as a format Implement BS as a part of their profiles for example the spdx and cyclon DX are also known

▶ 10:14 for esom and a c can be used as a security advisory uh let me break down open back's example so as I said back statement has three components the first is a vulnerability uh which can be expressed by identifiers like CB IDs the second is a product there are similar standards uh for representing software but open B recommends using package URL it's identifiers for pack software packages consisting of package ecosystem type like goang npm RPM also package

▶ 10:59 name space the package name like kubernetes and the version and the third is a status which can be affected uh not affected fixed and under investigation in open BS other formats have different sets of the statuses in this example we are looking at open back's statement about kubernetes and its sub component Docker distribution so the status is not set to not affect it which means kubernetes is using the vulnerable version of the DOA distribution but it's not affected by this vulnerability in this specific

▶ 11:41 Contex so this Contex really matters because the doer distribution itself has a vulnerability actually but in this context of kubernetes the varability doesn't affect so Contex to relationship really matters in BS so this is a typical uh way to use BS in our workflows first uh we detect vulnerabilities uh affecting our assets as I said by checking bability databases against asset information like esone then we reflect statuses from back

▶ 12:22 statements to filter out non exploitable vulnerabilities this way we only output vulnerabilities that are indeed exploitable so who can be so who can issue back statements the answer is simple uh anybody can back so uh there are various stakeholders in supply chain software supply chain and uh they can issue back statements for their own specific reasons uh let me give you a concrete example let's say uh vulnerability is found in ransy and the kubernetes

▶ 13:07 depends on ransy so this kubernetes maintenance review the source code and finds that kubernetes is not calling any of vulnerable functions in ransy in this case the kubernetes maintenance probably backs kubernetes I mean issue de statements for kubernetes the declaring kubernetes as not affected but it doesn't stop there like for example redut as a kubernetes distributor analyzes the the product and determine that the product is not affected by this vulnerability because the bable the vable functionality is not enabled in the product for example yes

▶ 13:51 in this case r can be kuet this also even end the user uh find like their usage is not affected by this pability then they can be kuet this but in this case probably they want to keep it internal to their organization because their end user it doesn't make sense to publish beex documents beex statements and also security researchers like the third party vendors researchers uh look into the kubernetes source code and discover that the kubernetes is not uh using ransy in a bable way so they

▶ 14:34 can B kubernetes so anybody anyone can beex and they can beex any products so beex worklow might sound straightforward first it beex is generated for a specific product then distribute it this carbon brought by users and finally apply to the detected vulnerabilities to get the final output yeah it sounds straightforward but in practice there are Challen similar challenges we have to address so one of the major challenges

▶ 15:18 is about distribution and Discovery so actually there are some more challenges in beex generation and beex application but in this talk I focusing on the distribution and the discovery problems so currently there is no standard for beex distribution which makes it difficult to share B documents effectively we might be able to use oci referers you know for B distribution in O registry for container images or oci artifacts but as types of software face this challenge so result standardized distribution methods users struggle to

▶ 16:02 find discover the relevant Vex documents because they simply don't know where to look so and the 3B uh bability scanner I maintain supports Vex through the dash dash Vex C plag which takes a local file past to the PEX document uh to filter out vulnerabilities and also some other scanners uh doing something similar but to be honest it's not practical because users need to find download and provide Vex documents to the scanner themselves but if your project has more than let's

▶ 16:46 say 100 100 dependencies which is quite common how do you find all the relevant V documents now it's impossible to manually track down all of them so this bring us to a crucial point to make Vex truly practical we have to automate both the distribution and the discovery Discovery processes so yeah be have uh yeah this is uh what I wanted to talk about today h we created be Hub to address uh the problems I just explained

▶ 17:29 uh bhub is a simple GitHub repository where beex documents are stored H when the software maintenance uh publish Vex documents they are automatically collected and stored in beex have it creates two a benefit like software maintenance can easily distribute beex documents through beex Hub and also users can easily discover beex documents from a centralized location so beex have makes it possible to automate beex Discovery uh we've already implemented

▶ 18:10 this automatic beex Discovery in triy and you can enable it using Dash dashb report C flag then during vulnerability scanning Tri automatically tries to find relevant Vex documents from vex have and apply them to the detected pabilities so in this example uh when T is scanning container image to be found binary built by go and also found found the beex document from beex H then suppress vulnerabilities accordingly as you can see so it set the suppressed viabilities

▶ 18:52 this varability was detected but uh suppressed because of the BS from Aqua beex Hub so you don't have to look for beex documents so yeah Tri it does and this is an example of how bab can be used in practice when the source code is built into the software artifacts like binaries or container images uh security scanners are often required to scan these artifacts without access to the original source code it's not easy to

▶ 19:35 analyze exploitability from the buildt artifacts you know from the binaries I know there are some researches regarding that but it's not easy so this is where our solution comes in when during the build process we can use static analysis tools like sast to generate BS H for example go has a tool called go Bon check that builds a core graph and evaluates exploitability and that tool also supports generating vs so then these beex documents are published to beex

▶ 20:18 sub and when security scanners examine the software artifacts they automatically discover and utilize Vex documents from Aqua vex Hub and I can show you how bhub collects be documents behind the scenes H beex Hub has a CER and its configuration is a managed in a gab repository to collect Vex documents for a specific product we add its package URL to the crer configuration in this example you can see three packages in this configuration

▶ 21:02 yeah actually the configuration has a list of packages to collect Vex documents so yeah in this case cow tries to collect Vex documents for these three packages next uh the cor tries to resolve the source repository URL from the package URL because the package URL is uh describing where this package is distributed but it doesn't to mention where this package is developed or where this package is built as you can see this in this example npm package just sets angular animations so we don't know where this package is developed so once

▶ 21:45 the corer gets the source report to URL ER it looks for back files under the dotb directory of the source repository URL when the crer finds beex documents beex files it stores beex files in aqua beex Hub uh this entire process runs you know periodically ensuring aquex Hub always has up toate beex documents so software maintenance uh don't have to notify aoex Hub about updates uh all they have to do is create beex document

▶ 22:29 and store that uh in theb directory of the source repository so if you a mainten of any open source repository so please consider publishing PEX documents it helps a lot of users with uh vulnerability Hing so this is an example of how a repository resolution Works uh in this example uh with the npm package angular animations uh looking at the bottom right of the slide uh you can see the repository information that uh the C Reves from the npm registry API so in this example it resoles to

▶ 23:13 github.com angular angular so it's just a resolution from angular animations npm package name to the GitHub repository URL once the corer gets a repository URL it the crws b files and a dotb directory this is an example from T repository as you can see there is a dotex directory at the root of the GitHub repository containing SAR beex files these beex files are collected by the crer and stored in beab like a copied into the uh vub so I should mention uh

▶ 23:59 we proposed this idea the use of the do tobex directory to the open beex project from open ssf but uh this is still under discussion so the future implementation might change based on this ongoing discussion so I said anybody can back anybody can back any product so for example while I'm not a kubernetes Mainer but I can back kubernetes you can B kubernetes anybody can back kubernetes so this naturally rais an important question so which BS should we trust it

▶ 24:40 so in aqua be have H we've implemented a trust model based on the maintainer authority it means AA beex Hub accepts beex documents for kubernetes that are issued by kubernetes maintenance so beex document that I issue or U issue wouldn't be accepted also Backs from third parties will be rejected Vex documents under the do beex directory have been added by project maintenance or accept it through Pro request they have reviewed so it gives us confidence that these B

▶ 25:22 documents represent the maintenance H official position but what if users want to trust be documents be statements from third parties more than on issued by project maintenance yeah actually it makes sense for example uh when a smaller project maintenance who is not familiar with security doesn't publish BS or even publishes wrong backs yes in this case you might want to trust Backs from third parties to address this need for flexibility we Define the B repository specification

▶ 26:07 which allows you which allows anyone to publish their own B rep poory so this spec is uh quite simple you know consisting of three components first is a manifest file containing metadata about the repository and the index file that Maps the package IDs to the corresonding Vex locations Vex file locations and third the collection of Vex documents also distribution is a very straight for uh you host the Manifest file on the server and you archive index file and B documents into a single archive file

▶ 26:49 like a Taj zip then distributed that's it so Aqua beex Hub follows this beex repos specification meaning just uh one of implementations uh for this spec and any repository uh trib supports any beex repository that implements this beex repository specification so it means you can choose whose backs you trust H whether that's from Project maintenance big tech companies or security researchers third parties so you can choose

▶ 27:32 and there are the examples of manifest file and index file uh manifest file contains metadata like repository name description uh a list of supported versions and the locations of archive files yeah so simple and index file contains a mapping between package IDs and the corresponding B file locations so v file locations are currently uh relative file pass within the archive file this structure allows you to allows security scanners to be efficient because security scanners first read the

▶ 28:14 index file and load only necessary back files for example if your project is not using npm package there's no reason security scanners load b files regarding npm packages so yeah I'm really honored to share that Rancher published their own B repository since uh Rancher bhub so this Rancher bhub uh follows implements the specification I just explained it's already compatible with 3B and any security scanners that support B report stories so for example uh this

▶ 28:59 repository distributes a beex document for kubernetes but this beex is not from kubernetes mainers so Aqua beex Hub will not accept this beex document but still s sua Rancher can distribute this B in their own B repository this is you know the one of the biggest benefits of the beex repository specification I explained so you can see there are required components uh manifest file index file and B documents it's a simple GitHub repository H trib uh allows you to

▶ 29:43 configure multiple repositories with priorities so for example you can set Rancher xub as primary and set up a xub as a secondary so when the B file for a specific product is not found in the primary repository then to fors back into the secondary repository so if you have like more than two repository so T falls back until the Vex document is found so you can also configure it the other way around set Aqua bab primary Rancher bab secondary or you can choose to use only Rancher xub it's totally up

▶ 30:27 to you so any beex repository that follows the specification I just explained can be used Tob and any scanners that support uh beex repositories uh yes all right so since uh we are seriously we've been seriously trying to reduce noise in vulnerability detection so for a long time so I'm truly happy to be here to today talking about our solution bhub and B repository specification so anybody can publish their own B reposer and also anybody can

▶ 31:10 and any security scanners can consume this be repositories so H I hope they will help make the world little more secure so yeah thanks for your attention and if you have any questions yes okay thank you oh oh yeah sorry uh so you said you guys use third party security Personnel to verify the Vex uh documents I guess is that the case um yeah you mean you are talking

▶ 31:53 about the be report story from third parties uh I guess so a I'm a vendor and I send you know I say here's my Vex um docu documentation whatever you want to call it and you're saying someone else verifies that after a vendor has sent it in or is it is it purely trust uh so you sorry you mean the you as the party security Benders parties you are you want to publish the be report story right so yeah basically you want to say I guess you want to you know if I use back I

▶ 32:36 want to know that the data use b mean the publishing BS or consuming B consuming it consuming spe okay so yeah you can just like configure like a b post like Tri for example Tri has a configuration file and you can point to any bory post three uh however you want so just uh like a if I know let's say I don't know red publish the V repository then you can configure it to the point to that xos to okay yeah so I guess what I'm trying to say is like how do I know that red hat isn't making up yeah things that's uh maybe I will we need to make a list of the Vex repository I don't know like index or

▶ 33:19 something maybe we have to make yeah yeah for now we don't have it so because it's it's uh launched a few month ago so only now luncher is uh publishing their re repository so now only two repos stories so okay yeah but hopefully it will be expanding yeah thank you hey um nice presentation we produce our Vex data so my question is that the Vex repository specification that you just mentioned is it under any open source like organization like what's the governance behind it yeah actually now uh we at Aqua security defined this specification uh but we are trying to

▶ 34:02 make it transparent and vender neutral so anybody can use it actually it's on GitHub so any repository any specification is a public so but if possible we want to you know donate this spec to like op ssf or something but I'm not sure so now for now it's defined by our company Aqua security yes thank so that was kind of my question and it was not necessarily a question but a request I would request that Aqua security donate the Vex repo specification in Vex Hub to op ssf so that op ssf could run Vex Hub as a public good instance just like they do with other sources of information essential for software

▶ 34:45 supply chain security yeah so that's the you know best so idea yeah so hopefully but I'm not sure like we were supposed to have a chat with open be Sig but we didn't have a chance so far okay so maybe after this I'll follow up with you later thank you thank you if I wanted to uh Implement Vex Hub support into a tool I maintain is there any sort of libraries or anything to support that or would it all be writing it from scratch right I was foring could you repeat it if I wanted to add Vex Hub support to a a tool I maintain or a product I maintain is there any sort of like tooling or libraries to help with that uh or is it I would have to yeah for now we don't have it sorry yeah so

▶ 35:30 for now there is just a specification and uh but yeah actually it makes sense maybe we probably make a you know GitHub organization to like help with Implement like consuming Vex repository or something yeah yeah I I was going to suggest some you know some sort of collaboration for like go open Vex python open Vex you know actually the open BS have one repository like open B Discovery Repository and I was trying to add support for poster in that library but yeah so far we didn't have a chance so probably we were working together to out that support

▶ 36:14 yeah hi thank you excellent work um I had a question about Vex Hub uh it seems like an excellent source but it's only going to be as good as it is as good data so knowing that it's still new how complete is the data in there do you have a lot of data or a little bit of data or how are you going to get it fully populated with as much data as possible sry could you elaborate on that so if if Vex Hub only has a handful of of Vex documents in there it's only going to be very limited in its usability right of trying to build secure software so my question is how how big is your data set already is it already usable for the average company to start adopting or is this sort of

▶ 36:58 hey this may only get 1% of the software you're using yeah so you mean the current vub has like a small set of the Vex documents so but uh you mean yes my question is how how big is the data set that Vex Hub has today yeah so for now we just need to promote beex itself and uh if the software maintenance publish beex documents it helps a lot of you know users with vulnerability handling so we just need to say please publish beex documents and yeah for now actually beex itself is not so popular no as of today so yeah but you know cisa is also working on BS and also op ssf is

▶ 37:41 trying to promote BS so yeah now uh hopefully after a few years the most open source projects so vex vex Hub is a little more aspirational at the moment than it is usable yeah okay thank you well I appreciate your work yeah thank true