Assessment vendor selection checklist: scored RFP, POC acceptance criteria, and contract red-lines

Most assessment vendor decisions fall apart in the same place: the gap between what the sales demo showed and what the platform actually does when your own items, your own candidates, and your own integrations hit it. The demo always works. The POC is where you find out the equating module can't handle your form structure, or that "SSO support" means SAML but not the specific IdP your IT team runs.

A vendor selection checklist that lives in a spreadsheet of yes/no questions doesn't catch any of this. What catches it is three connected things: a scored RFP that forces real comparison on the stuff that matters, a POC with pass/fail acceptance criteria written before the trial starts, and a short list of contract clauses you won't sign without. This post is about building all three so they reinforce each other instead of sitting in separate documents nobody reconciles.

Why the typical RFP produces vendors that all look identical

There's a pattern here that repeats constantly. A team sends out an RFP with 120 questions. Every vendor answers "Yes, fully supported" to about 95% of them. You end up with four spreadsheets of yeses and no meaningful way to differentiate, so the decision quietly collapses back onto price and whoever the loudest stakeholder liked in the demo.

The problem isn't the number of questions. Binary questions don't discriminate. "Do you support item response theory?" gets a yes from a vendor that ran IRT once in a research paper and a yes from a vendor whose entire calibration pipeline is built on it. Same answer, wildly different reality.

1. 1
   Make questions evidence-based, not capability-based. Instead of "Do you support DIF analysis?" ask "Attach a sample DIF report from a live client program (anonymized) and describe the detection method and flagging thresholds." Now the yes has to come with proof.
2. 2
   Weight and score by category, not by count. A psychometrics question that determines whether your scores are defensible should not carry the same weight as whether the UI has dark mode.

The second you attach weights, the conversation inside your own team changes. People have to argue about what actually matters before they see the vendors, which is exactly when that argument is useful.

Building the scored RFP: weights that reflect real risk

Split your RFP into weighted categories. The weights below are a starting point for a mid-size program running high-stakes or compliance-relevant assessments. If you're running low-stakes formative quizzes, shift weight away from psychometrics and toward usability and content authoring.

Each question inside a category gets scored 0–3, not yes/no:

1. 1
   0 — Not supported, or supported only with custom development you'd pay for
2. 2
   1 — Supported but with meaningful limitations or manual workarounds
3. 3
   2 — Supported natively, works as described
4. 4
   3 — Supported natively and backed by evidence (client reference, sample output, documentation link)

The 3 tier is the one that separates real vendors from pitch decks. Most answers that would have been a confident "yes" land honestly at a 1 or 2 once you require evidence.

Psychometrics questions that actually discriminate

1. 1
   Which calibration models are supported natively (CTT, 1PL/2PL/3PL, partial credit, graded response)? Attach a sample calibration output.
2. 2
   How does the platform handle equating across forms, and what methods are built in versus requiring export to external tools?
3. 3
   What's the minimum sample size the system recommends before producing stable item statistics, and does it warn users below that threshold?
4. 4
   How are flagged items (poor discrimination, DIF, high omit rates) surfaced to reviewers? Screenshot the actual workflow.
5. 5
   Can scoring rules and cut scores be version-controlled and audited?

If a vendor can only answer the first bullet and goes vague on the rest, their psychometrics is a feature, not a foundation.

Security and integration questions worth their weight

Security questions should never stop at "Are you SOC 2 compliant?" A current SOC 2 Type II report is table stakes — ask for the report itself under NDA, and ask when the last one was completed. Then go operational:

1. 1
   Where is data physically stored, and can you pin storage to a specific region?
2. 2
   What's the encryption posture at rest and in transit, and who holds the keys?
3. 3
   How is access to candidate PII logged, and can you get those logs?
4. 4
   What's the data retention and deletion policy, and can you trigger deletion on demand?

For integrations, the single most valuable question is about leaving: "Describe your full data export. What formats, which fields, and does it include item statistics, response-level data, and audit logs — or just score summaries?" Vendors who make it easy to arrive and hard to leave tend to reveal themselves here.

POC acceptance criteria: write the pass/fail before the trial starts

A POC without pre-written acceptance criteria isn't a test. It's a guided tour where the vendor steers you toward what works and away from what doesn't. The fix is simple and almost nobody does it: define what "pass" means in writing, get both your team and the vendor to agree to it, before any data touches the system.

Acceptance criteria should be binary and observable. Not "the equating feature works well" but "equating two 40-item forms using our actual field-test data produces comparable scale scores, and the output matches our independent calculation within an agreed tolerance." Someone can look at that and say pass or fail without debate.

A workable POC acceptance structure:

1. 1
   Load real data, not sample data. Import a slice of your actual item bank — including the messy items with incomplete metadata. If the import chokes on your real content, you want to know now, not after signing.
2. 2
   Run one full workflow end to end. Author or import items, assemble a form, deliver to a small test group, score, and pull reports. Every handoff between those steps is a place things break.
3. 3
   Test the integration that matters most. Pick your highest-risk connection — the SSO, the HRIS roster sync, the LMS gradebook push — and make it work with your actual environment, not a sandbox the vendor pre-configured.
4. 4
   Break something on purpose. Submit a malformed roster, a duplicate candidate record, a response with special characters. See how the system fails and whether it tells you clearly or silently corrupts.
5. 5
   Time the support response. Open a real support ticket during the POC and measure how long it takes to get a useful answer. This is the single best predictor of what support actually feels like after the contract is signed.

For each criterion, write it as a sentence you could check off. Then score the POC on those criteria only. Scope creep during a POC — "oh, we can also show you this other thing" — is usually a sign that the agreed criteria aren't going well.

“

Have both your technical lead and your psychometrician sign each acceptance sentence to avoid later disputes.

A short POC acceptance checklist

1. 1
   [ ] Our real item bank slice imported without manual cleanup beyond documented, acceptable steps
2. 2
   [ ] A full authoring-to-reporting cycle completed by our staff, not the vendor's
3. 3
   [ ] The priority integration passed a live test against our environment
4. 4
   [ ] At least two deliberate error cases were handled gracefully and visibly
5. 5
   [ ] Psychometric output on our data matched our independent check within tolerance
6. 6
   [ ] A support ticket was resolved within the SLA they claimed
7. 7
   [ ] Data export produced response-level data, not just summaries

If more than one or two of these fail, the issue is rarely something a "future roadmap" fixes in time for your go-live.

A real scenario: the certification program that caught it in the POC

A professional certification body — small team, roughly 6,000 candidates a year across two exam cycles — was replacing an aging platform. Two finalists scored within a few points on the RFP, both strong on authoring and reporting. On price, the cheaper one came in around $18k/year under the other.

Instead of letting price decide, they ran identical POCs with pre-written acceptance criteria. The cheaper vendor's platform handled authoring and delivery well. But when the team loaded their actual two-form structure and ran equating, the output drifted meaningfully from their independent calculation. The vendor's answer: "advanced equating" required a professional-services engagement billed separately — somewhere in the $12k–$15k range per cycle.

That turned an $18k annual saving into a net loss. And more importantly, it would have surfaced only after signing if they hadn't tested equating on real data. They went with the pricier vendor. The POC paid for itself before the ink dried.

The lesson isn't "spend more." It's that the thing that would have sunk the program never appeared in either RFP response — both said yes to equating. It only showed up when real data hit a pre-written pass/fail test.

Contract red-lines: the clauses you don't sign without

By the time you're in contracting, the leverage you had during the sales cycle is quietly draining away. The vendor knows you've invested months and have an internal go-live date. This is exactly when weak clauses get waved through. A short list of non-negotiables:

1. 1
   Data ownership and export on exit. You own your data — items, responses, statistics, logs — and the contract guarantees full export in a usable format within a defined window at termination, at no extra charge. Watch for "data" being defined narrowly enough to exclude item statistics.
2. 2
   No silent model or scoring changes. If the platform uses automated scoring or any AI-assisted components, the vendor must notify you before material changes to models that affect scores, and provide documentation. This ties directly to defensibility; if a scoring approach shifts mid-cycle without disclosure, your cohort comparisons break and you won't know why. Our piece on responsible AI governance for assessments covers the operational guardrails worth referencing in this clause.
3. 3
   Breach notification timeline in hours, not vague "promptly." Specify a hard number — 48 or 72 hours — and require notification to include scope and affected records.
4. 4
   SLA with real teeth. Uptime commitments mean nothing without service credits and an exit right if the vendor misses the SLA repeatedly over a defined window.
5. 5
   Price protection on renewal. Cap renewal increases. Open-ended renewal pricing is how a reasonable year-one deal becomes an unaffordable year-three one.
6. 6
   Pass-through of professional-services pricing. If features like equating or custom reports require paid services, lock those rates into the contract now, while you still have leverage.
7. 7
   Subprocessor transparency. You get a list of subprocessors and notice before new ones handling your data are added.

The red-line that gets skipped most often is the export-on-exit clause, because at signing nobody wants to think about leaving. That's precisely why it matters. The cost of switching vendors later is almost entirely determined by whether this clause exists.

When this level of rigor makes sense — and when it doesn't

This whole process has real overhead. A scored RFP, parallel POCs, and contract red-lining can stretch a selection cycle by a couple of months. That's worth it for high-stakes, compliance-relevant, or long-horizon decisions. It's overkill for a low-stakes formative tool you could swap out in a weekend.

When the full rigor is worth it:

1. 1
   Scores feed hiring, certification, licensure, or promotion decisions
2. 2
   You're signing a multi-year contract with meaningful switching costs
3. 3
   Candidate PII and data residency carry legal exposure
4. 4
   Your psychometric defensibility could be challenged

When to run a lighter version:

1. 1
   Low-stakes, internal-only assessments
2. 2
   Short contracts with easy exit
3. 3
   Small candidate volumes where a bad fit is cheap to correct

Who should not skip the POC entirely: anyone whose program depends on integrations or equating. Those are the two areas where "fully supported" and "actually works on your data" diverge most often, and they're exactly the areas a demo is designed to gloss over.

Tying the three documents together

The mistake even careful teams make is treating the RFP, the POC, and the contract as three separate exercises owned by three different people. They should be one chain. The weighted RFP tells you what to test in the POC. The POC results tell you which contract clauses actually need hardening. A vendor who scored a vague "1" on equating in the RFP and then failed the equating POC criterion is a vendor whose contract needs professional-services pricing locked down — or no contract at all.

Here's a simple diagram of the workflow to keep the chain intact.

Keeping that chain intact is largely an organizational discipline problem, the same kind that shows up when assessment programs grow faster than their processes. If your team already struggles with who owns what across the selection, the broader fix in our write-up on governance when assessment operations scale applies here too: define the roles and handoffs before the vendor process starts, not during it.

Vendor selection goes wrong not because teams ask too few questions, but because they ask the wrong kind — binary, unweighted, untested, and unenforced. Score the RFP so vendors can't all look the same. Write the POC pass/fail before you're inside the vendor's guided tour. Red-line the contract while you still have leverage, not after your go-live date has been announced. Do those three in sequence and the demo-to-reality gap stops being a surprise you discover in production.