Assessing Vendors

A Hands-On Guide to Assessing Infosec and IT Vendors

Josh More

• Publisher: Syngress
• Year: 2013
• ISBN:
  9780124104464
• Topic: Negocios y empresa
• Subtopic: Inteligencia empresarial

Assessing vendors is a tricky process. Large and regulated organizations are forced to demonstrate due diligence in vendor assessment, but often do not know how to do this. This results in a great deal of busywork being required by both the vendors and the organizations. Smaller organizations don't know what to look for and, as a result, often wind up selecting based on price instead of value. This results in service failures and vendors that just milk their customers for as long as they can.

Assessing Vendors shows you how to walk the line between under- and over-assessing, so decisions can be made on sufficient data without wasting time, digging too deeply, or making decisions too quickly. This hands-on guide will show you how to use an iterative approach to vendor analysis, so you can rapidly filter out the vendors that are clear failures and then select likely winners. It will then show you how to do progressively deeper dives into the likely winners so you can select a preferred vendor. Finally, you will learn how to negotiate with your preferred vendor to get reasonable prices and services.

• Provides an iterative approach to vendor assessment, showing you how to make decisions on sufficient data without wasting time
• Includes checklists to help you navigate the decision-making process, while considering all the important factors needed to make a sound decision
• Helps you understand and evaluate vendors based on key concepts such as performance criteria, functional testing, production, and price

• Provides an iterative approach to vendor assessment, showing you how to make decisions on sufficient data without wasting time
• Includes checklists to help you navigate the decision-making process, while considering all the important factors needed to make a sound decision
• Helps you understand and evaluate vendors based on key concepts such as performance criteria, functional testing, production, and price

Phase 1

Preliminary Research

1.1 Preliminary Research

1.1.1 Identifying Vendorspaces

1.1.2 Identifying Candidates

1.1.3 Building Criteria

1.1 Preliminary Research

1.1.1 Identifying Vendorspaces

Vendors tend to create spaces for themselves in markets. This makes it easier for prospects to find them, but it also restricts the way in which you think about them. A vendor solidly branded in one “vendorspace” is less likely to be considered outside of it, regardless of how well they may fit other needs. This is made worse by analysis firms who like to say things such as “Vendor X should pick a market and dominate it instead of trying to be everything to everybody.” So, while vendorspaces can be very useful in identifying candidates, keep in mind that the goal in selecting a vendor is to meet a business need, not to pick the “best” candidate on someone else’s list.

In most cases, you will be searching within a well-known vendorspace, such as “disk encryption vendors” or “intrusion prevention vendors.” However, vendorspaces have life spans. For example, text-based word processing tools were once incredibly popular, however, that vendorspace has contracted and the winners, Microsoft Word and WordPerfect, were migrated into graphical user interfaces. Along the way, tools like AppleWorks and WordStar died out. Thus, if you are exploring a vendorspace that is near the beginning or end of its life cycle, your choices within that space will be more limited and you may need to look further afield to solve your problem. Knowing this will help you identify when you have a reasonably complete list of candidates.

Vendorspace lifetimes vary with the type of solution and in relation to other competing vendorspaces.

Vendorspaces near their end are fairly easy to identify. Any technology that is referred to as “old school,” “legacy,” or “dead” likely reflects a contracting vendorspace. This isn’t necessarily bad; after all, there are still a lot of people using legacy mainframe technology. However, there will likely be few newcomers in those spaces, so you can shortcut this phase of the process as there will be less discovery involved.

Vendorspaces near the beginning of their rise are harder to work with. Not only are there not many vendors in them, but there will be significant disagreement between vendors as to whether or not they are in the space and whether that space even exists. At the time of this writing, the Unified Threat Management (UTM) and Next Generation FireWall (NGFW) vendorspaces have effectively merged and a new “Intelligent Threat Management” (ITM) vendorspace may be beginning to emerge. This means that while vendors such as Fortinet, Sophos, Watchguard, and PaloAlto fight with one another for dominance over the merged UTM/NGFW vendorspace, other vendors such as Barrier1 and MetaFlows choose not to compete there and instead are creating the new ITM vendorspace. This happens because, as these firms provide innovative approaches to problems, they are not exactly NGFWs or UTMs and may look fairly rough when compared to polished examples from those vendorspaces. However, because their own vendorspace hasn’t fully materialized yet, their new capabilities aren’t truly understood by prospective customers.

The primary rule in vendor assessment and selection is to always keep your needs first in mind. Then, if one vendorspace doesn’t clearly solve your problems, you will have to consider several and look to the overlap. Before you can start exploring, you’ll need a map. To make this map, consider all ways that your problem can be solved and build a list of possible solution types. This will help guide you to specific vendorspaces, so you can decide which ones you will be focusing upon.

1.1.2 Identifying Candidates

The first step in any assessment process is to try to gather as many candidates as possible. Looking at vendorspaces can be a good guide for candidates, but it should not be the only one. Because the assessment process operates as a funnel, continuously filtering out unacceptable options, the more options you put in the top, the more certain you will be of the final result. It is important, in doing this, to not limit yourself to only a single solution class. If your business is hostile to open source, those solutions may not be ideal. However, if they work for your purposes and none of the proprietary options do, they might wind up being the best choice from a set of imperfect solutions. You should also consider options that are either partial or overly-complete. Many “suite” solutions do far more than you might need to solve your problem, but this isn’t a reason to exclude them. This is a point of price negotiation later in the process.

To develop the full list, you must pull from many sources. The so-called “independent” analysis firms tend to use vendorspaces to compare technologies. Thus, once we’ve identified the name the market assigns to the type of solution we need, we can search for lists of options in that space. For example, suppose you were looking to compare Intrusion Prevention Systems, your first place to check may be Wikipedia. Then you can check for lists from Gartner, Forrester, and IDC. Most of these lists are pay-only, but the firms that score well tend to promote the fact in their press releases, so even if you can’t get the report itself, you can often get the list of the firms on the report. Press releases can be searched at http://www.pressreleasespider.com/.

Give yourself 5 minutes to build a list from Wikipedia, then 10 minutes each to search out vendors in the space from each of the top three analysis firms.

A Note on “Independent” Analysis Firms

As you do this step of the analysis, do not fall into the trap of thinking that these big-name research firms are the final word in the space. Remember, all you are doing is identifying candidates.

These firms require a lot of money to operate and, because of this, they charge vendors to participate in their studies. Some of them even have a “preassessment” service that lets the vendors pay even more money to help the analysis firms set up the technology to maximize their score in the tests. Because of this, you cannot trust that the list is complete (some vendors choose to not participate) and you cannot trust that the scores are fair (as some vendors do not pay for preassessment help).

Additionally, as the assessment firms are focusing on a general case, there is no way that they can know your particular needs, so even if the list were complete and the score were accurate, there is no way to tell how well it would apply to what you need. Thus, these reports are good sources from which to build lists as well as being good sources of concerns about a technology or business. They are not necessarily good places from which to pull your final choice.

Once you have your initial list of names, it’s time to fill in further. This is where search engines come in. A good way to do this is to actually look at the ads. Vendors who want to enter a market will pay for online advertising, so if you just search on a topic and turn off your ad-blocker, you can fill in your list.

As an example, suppose you are searching for an Intrusion Prevention System, you need to build a list of keywords. Luckily, Google makes this easy for you. Go to their keyword tool (https://adwords.google.com/o/KeywordTool) and search on “IPS,” “Intrusion Prevention Systems.” This feeds back 100 keyword terms, including “intrusion prevention software,” “ips monitor,” and “network intrusion detection.” Go through the list and pull out all the ones that seem to work well. Keep the list manageable. At this stage, I don’t like to go over 20 terms, but your numerical limit may be different. Selecting too few terms results in not feeding enough candidates into the process. Selecting too many means that you spend more time than is necessary doing searches. If you don’t have your own rough limit yet, a good rule of thumb is to start at one term and keep doing searches until you do several searches in a row that don’t identify new candidates. Your numerical limit will be lower than that.

For example, at the time of writing this, the tool suggested the following:

Clearly, some of these, such as “firewall” and “network security” are too general to be useful. However, this search alone points out that these devices tend to come in three flavors: Network, Wireless, and Host-based, that there are free options that could be considered, and that Cisco, Sourcefire, and Symantec are likely established and well-known vendors in this vendorspace.

Once you have the list, do a search on each term on an ad-supported search engine such as Google and Bing. Glance through the first few pages looking at both the results and the ads and fill in your list of potential vendors. This step in the process should take no more than an hour.

Extending this search on Google and Bing for the terms “Intrusion Detection System,” “Intrusion Prevention System,” and “NIDS” returns the following list of advertising vendors to consider:

Some of these, of course, are likely false positives and reflect resellers of solutions and not the solutions themselves. However, those will be rapidly filtered out in the following steps. Like a brainstorming exercise, the goal here is to gather as many potentially useful names as possible.

Now, you’ve spent less than two hours and should have a reasonably complete set of names for future analysis. If you wish to go further, there are three optional methods that you can use to fill in the list further. As they involve one-on-one communication, these take additional time, but can do wonders if your list feels somewhat thin.

The first method is to pick a company you recognize and contact a sales person there. When they answer, tell them that you are considering their solution, but you have to look at two competing offerings. Ask what else you should consider. In this instance, the sales person will want to drive you toward solutions that they consider less good than theirs—after all, they want the sale. What’s nice about this approach is that you are likely to get names that won’t be on the analysis reports, as products that fail the “independent” tests don’t tend to trumpet that fact on the Internet. If the reason for failure doesn’t apply to your use case, it doesn’t matter to you. The downside of this approach, of course, is that it gets you on the sales person’s call list. While some will just contact you a couple of times and give up, others will hound you forever. It may be wise to use throw-away e-mail addresses if you take this approach.

The second method is to go to a user group or association, call a friendly competitor, or talk to other people in your area and just ask what you should consider. Most technical people have a mental list of cool tools they always wanted to play with but couldn’t. These may not be suitable to your needs (or theirs), but as the goal is to fill in your list, this is a valid approach. You can often get more cutting/bleeding edge tools on your list this way.

The third method is something of a hybrid. Odds are that there are some Value-Added Resellers (VARs) in your area. Often, the technology that you are considering is resold by a VAR. Call each one of them, explain what you’re doing, and ask if they have any vendors that should be on your list. Most VARs are friendly and will give you a list of technologies that they like. Sure, in th...

Table of contents

• Cover image
• Title page
• Table of Contents
• Copyright
• Acknowledgments
• Introduction
• How to Read This Book
• Phase 1. Preliminary Research
• Phase 2. Sort
• Phase 3. Functional Testing
• Phase 4. Scoring, Weighting, and Sorting
• Phase 5. Deep Testing
• Phase 6. Adjusting Needs
• Phase 7. Negotiating Price
• Phase 8. Production
• Phase 9. Conclusion