Reducing risk in procurement bid scoring: Generate bespoke scores for your grade or rating lists

For nearly all procurement professionals, it's tough to produce bespoke tender evaluation systems again and again. After painstaking work defining criteria and producing criteria weightings, most would then go with a copy-and-paste set of “grade” or “rating” scores such as this:

Dangers with off-the-shelf set of grade scores:

• Not aligned to your needs
• Wrong bidder incentives
• May select a less aligned bidder

But what if you could determine your criteria weightings far faster? And what if there was an easy to understand and implement approach to produce a tailored set of grade scores?

Risks resolved by tailored set of grade scores:

• Incentivise bidders correctly
• Select the most aligned bidder
• Tried and tested method
• Easy to understand and do

Read on for one method that does both.

The most common approach to procurement bid scoring is unnecessarily risky

When determining how bids will be scored during a medium to large procurement exercise, the most common method is to have a set of grades or ratings (A, B, C … / blue, green, amber …) that can then be assigned to each piece of criteria for each bidder.

These grades each have a numerical score (100, 80, 60 …), which decides the destiny of the whole procurement. However, it is common for procurement teams to use a simple, one-size-fits-all set of grades and scores for many different contexts and contracts.

Those grades and scores might look like this:

…or this

… or anything else. But, whichever is chosen, if you are using one generic “grade score profile”, you are sending unintended messages to the bidders and silently rigging the outcome of the procurement.

These score profiles, together with the weightings of each of your criteria, are the entire means of incentivising the bidders to focus on the right (or wrong) things in their bid, and they ultimately select the winning bidder. A generic profile will incentivise bidders incorrectly, it may select a bidder that is not the most well-aligned to your needs, and it even invites bidders to game your score system. If that sounds far-fetched, it really isn’t, I have another blog post coming soon that gives the reasoning together with a full worked example so you can see for yourself.

But all of this risk is unnecessary.

There is a way to produce criteria weightings more quickly in order to free up your time. And, there is a way to easily produce robust grade score profiles that precisely align with your needs.

The solution? Easily produce robust weightings and grade scores

If your first response is scepticism, then let me start by telling you where it came from (and you’ll see why it’s a big deal).

In the 1960s, Thomas L Saaty observed that even the world’s best scientists and lawyers were struggling to negotiate international weapons disarmament (more complicated than your average procurement). At the heart of it was that they needed an easy way of converting masses of conflicting priorities into a single list of needs, where the importance of each need was represented by a numerical score. Starting to sound familiar?

He came up with the analytic hierarchy process (AHP), which is very simple but has proven to be phenomenally effective in an increasingly wide range of disciplines. For over a decade it has been the most commonly referenced method of its kind, and the company Saaty founded now provides AHP services to an impressive breadth of clients.

The 6-step analytic hierarchy process for procurement bid scoring systems

The process is documented all over the web to varying degrees of depth. What follows is an adaptation of the analytic hierarchy process to the precise area of grade profiles for procurements. In fact, AHP is also very effective in deciding each weighting for each of your criteria (e.g. 15.7% for “Service Delivery” 12.2% for “IT Solutions”, etc), so I’ve referred to both.

If you use the process for criteria weightings as well as score profiles, start with criteria weightings, then run through it again for score profiles. Here it is:

Step 1:

Ensure you have a clear and appropriately detailed definition for each piece of criteria / each grade (note that this is necessary whether or not you are using AHP).

When defining grades for a grade score profile, make your definitions as specific as possible to the piece / pieces of criteria (“Service Delivery”, “IT Solutions” etc) to which the score profile refers.

Step 2:

Each Subject Matter Expert (SME) compares just two of the criteria or grades, say “good IT Solutions” and “very good IT Solutions” (they will move onto other pairs in step 4), taking into account the detailed definitions from step 1.

Step 3:

They each decide how different these criteria or grades are by selecting from a list of options or using sliding scales such as those in the screenshots below, which assigns a number of points to them.

Different ways each AHP decision could be made:

The points from each SME can be automatically combined using four simple equations. This means that, even after the SMEs have only each made one decision, it is already possible to visualise precisely how much more important “very good” is deemed to be than “good”.

Step 4:

Repeat this for more pairs of criteria or grades.

Before you’ve even finished, you will be looking at how the entire set of criteria weightings / score profile is shaping up and which decisions the SMEs find hardest to agree on.

Step 5:

…and so on, reviewing decisions that are most inconsistent with others, until you have enough confidence in your weightings or score profile(s).

An optional but powerful part of step 5 is to bring SMEs together in a workshop to see precisely (thanks to AHP) the areas in which they most disagree, and discuss their reasons more succinctly than would normally be possible.

This can sharpen up what each grade or piece of criteria actually means, and bring SMEs’ varying priorities out in the open in a way that wouldn’t have happened otherwise. Often this is so effective that the SMEs will form a consensus there and then on the basis of this new information.

Step 6:

If you have completed steps 1 to 5 for criteria weightings (i.e. not grade score profiles), then you must now decide what grade score profile(s) you need, then repeat steps 1 to 5 for the grade score profile(s).

If you decide to have multiple different grade score profiles for your procurement (e.g. you might produce one for “Service Delivery”, another for “IT Solutions” etc), then you can have fewer grades in each score profile and still end up with a more precise scoring system overall. For example, you might go from having one 7-grade A-G profile to a mixture of 4-grade, 5-grade and 6-grade profiles. While the optimal approach is one profile per piece of criteria, it is up to the procurement team to decide how close to the optimal approach they should go.

… and that’s it. You were waiting for the really complicated black-box modelling… well, there isn’t any. And yet, what I’ve just described is powerful enough to notably simplify international weapons disarmament.

Between 4 and 10 decisions per person (your choice), coupled with a workshop in which disagreements are precisely identified and defined, is enough to produce a tailored score profile like the one below, that aligns with the needs and opinions of all of your subject matter experts.

Yet more benefits of AHP for procurement

• These simpler “pair-wise” decisions are easier to legally defend & less prone to bias than other methods of tailoring weightings or score profiles
• The number of different score profiles is up to you (obviously more profiles results in more precise alignment with your goals)
• The increased precision of your scoring means that you don’t need as many grades in each score profile (which would make a good future blog post, though please contact us if you’d like to hear more before we’ve written it!)

There are a plethora of websites containing the theory as well as examples, so much so that many multi-discipline teams already have the expertise to set themselves up in this. Do get in touch though if you want to move more quickly or to explore further how powerful this process could be for you.