Assessment interoperability standards mapped to procurement acceptance tests

Most procurement failures with assessment platforms don't happen at the demo. They happen three months after signing, when your item bank won't import cleanly, single sign-on breaks for one department, and the analytics dashboard you were promised turns out to be a pile of experienced xAPI statements nobody can query into anything useful.

The vendor said they were "QTI compliant." They said they "support LTI." They said everything "flows to your LRS." All technically true. None of it tested against your actual data, your actual identity provider, or your actual reporting needs.

This article is about closing that gap — not by trusting the RFP checkbox, but by writing acceptance tests that force the vendor to prove interoperability against your environment before money moves. If you're an educator, HR lead, or corporate trainer sitting across from an architecture team and a sales rep, this is the layer of scrutiny that separates a clean rollout from a two-quarter integration nightmare.

Why "standards compliant" means almost nothing without a test

The three standards you'll hear most often each solve a different problem, and vendors love to conflate them:

1. 1
   QTI (Question and Test Interoperability) governs how assessment content — items, tests, response processing, metadata — is packaged so it can move between systems.
2. 2
   LTI (Learning Tools Interoperability) governs how your LMS launches an external tool, passes identity, and receives grades back.
3. 3
   xAPI (Experience API) governs how learning and assessment activity gets recorded as statements into an LRS (Learning Record Store).

The catch: all three have versions, profiles, and optional features. "QTI compliant" could mean QTI 2.1 with only multiple-choice support, or QTI 3.0 with full response processing and accessibility metadata. "LTI support" could mean LTI 1.1 (deprecated, weak security) or LTI 1.3 with Advantage services (Names and Roles, Assignment and Grade Services, Deep Linking). "Sends xAPI" could mean it emits three statement types with no result scoring and a broken actor identifier.

What shows up repeatedly across platform evaluations is that vendors answer the category question honestly and dodge the profile question entirely. Nobody's lying exactly — they just answer "do you support LTI" instead of "do you support LTI 1.3 Assignment and Grade Services with line-item creation from Deep Linking." Those are completely different levels of maturity, and only one of them actually sends grades back to your gradebook automatically.

The entire game is translating vague claims into specific, testable assertions.

The mapping: claim → standard → acceptance test

Bring this table to the vendor call and refuse to accept a "yes" without a matching test result.

The right-hand column is where procurement actually happens. Everything else is marketing.

QTI: the round-trip test nobody runs (and always should)

The single most useful QTI test is a round-trip: export a package from your current system (or a reference package you build), import it into the candidate platform, then export it back out and compare.

Content that imports but degrades is the classic failure. In practice this usually looks like: multiple-choice items come through fine, but matching items lose their scoring logic, extended-text items lose the rubric association, and item metadata — difficulty, competency tags, accessibility flags — silently drops. The platform "imported successfully." It just quietly discarded the parts that make your bank valuable.

A practical QTI acceptance script:

1. 1
   Prepare a known-answer package of around 20 items covering every interaction type you actually use (not every type QTI defines — just yours).
2. 2
   Include items with media (images, audio), items with response processing (partial credit), and items with rich metadata.
3. 3
   Import into the candidate platform.
4. 4
   Render each item and confirm it displays correctly — check hotspot coordinates, drag-target zones, audio playback.
5. 5
   Score each item with a deliberately wrong answer, a partially correct answer, and a fully correct answer. Confirm scores match expected values.
6. 6
   Export the package back out.
7. 7
   Diff the exported package against the original. Flag any lost metadata, altered scoring, or dropped media references.

If step 7 shows silent loss, you now have a concrete defect list instead of a vague bad feeling. That's real leverage in a negotiation, and it tells your team exactly how much cleanup migration will cost.

One pattern worth naming: platforms that are strong on authoring are often weak on import fidelity, because they'd rather you build content in their tool than bring your own. That's a business incentive baked into the software, and it directly affects how portable your item bank stays. If you've spent real effort retrofitting legacy item banks and tagging them for analytics, a lossy importer erases that investment on day one.

LTI: prove the grade actually comes back

LTI is where the demo lies most convincingly, because a launch looks like integration. The tool opens inside the LMS, the learner's name appears, everyone nods. But the launch is the easy part. The hard part is Assignment and Grade Services actually posting a score back to the right line item without a human clicking "sync."

1. 1
   Launch test. Confirm the launch uses LTI 1.3 (OIDC + signed JWT), not a 1.1 fallback. Inspect the launch payload. Confirm the role claim maps correctly — an instructor should not launch as a learner, and vice versa.
2. 2
   Deep Linking test. Have an instructor select a specific assessment from the LMS. Confirm a line item is created automatically in the gradebook. If the instructor has to manually create a matching column, deep linking isn't really working.
3. 3
   Grade pass-back test. Have a test learner complete an attempt. Confirm the score posts to the correct line item, with the correct maximum points, within the stated time. Then re-take and confirm the score updates rather than creating a duplicate.
4. 4
   Roster test (NRPS). Add and drop a learner in the LMS. Confirm the tool's membership reflects the change within the SLA the vendor promised. Silent roster drift is a top cause of "why didn't this student get their assessment" tickets six months in.

A sample AGS score payload should look roughly like this — and you want to see the real thing from a captured session, not a slide:

``json { "userId": "a7c3-9f21-4b8e", "scoreGiven": 78, "scoreMaximum": 100, "activityProgress": "Completed", "gradingProgress": "FullyGraded", "timestamp": "2026-03-04T14:22:07.000Z" } ``

If scoreMaximum doesn't match your line item, or gradingProgress never leaves Pending, your gradebook will show blanks and nobody will know why. Ask to see this JSON. A vendor who can't produce a captured payload in ten minutes probably hasn't tested AGS in your configuration.

xAPI: the statements that look fine and report nothing

xAPI is the standard most likely to pass a surface check and fail every real analytics question. The reason is that xAPI is permissive by design — almost anything valid can be a statement — so "we send xAPI" is technically true even when the statements are useless for reporting.

1. 1
   Inconsistent actor identity. If the same learner appears as an email mbox in one statement and an account object in another, your LRS can't group their activity. Every "learner journey" report falls apart.
2. 2
   Missing result detail. Statements that record completed but no score, duration, or success flag tell you that something happened but nothing about how it went.
3. 3
   No context grouping. Without a registration or context.contextActivities, you can't tie a set of statements to a single attempt, cohort, or course. You just get a flat stream of events with no structure.

``json { "actor": { "account": { "homePage": "https://id.yourorg.com", "name": "a7c3-9f21-4b8e" } }, "verb": { "id": "http://adlnet.gov/expapi/verbs/scored", "display": { "en-US": "scored" } }, "object": { "id": "https://assess.yourorg.com/items/safety-201/attempt/5512", "definition": { "type": "http://adlnet.gov/expapi/activities/assessment" } }, "result": { "score": { "raw": 78, "min": 0, "max": 100 }, "success": true, "completion": true, "duration": "PT12M34S" }, "context": { "registration": "b91f-22a0-77cd", "contextActivities": { "grouping": [{ "id": "https://assess.yourorg.com/courses/safety-201" }] } } } ``

The xAPI acceptance test:

1. 1
   Run a full assessment attempt end to end.
2. 2
   Query your LRS (or the vendor's) for every statement generated by that single attempt.
3. 3
   Confirm you can reconstruct the entire attempt — who, what, score, duration, pass/fail, which cohort — from the statements alone, with no help from the platform's UI.
4. 4
   Confirm actor identity is consistent across all statements for that learner.
5. 5
   Run the same for two learners in the same cohort and confirm you can produce a cohort-level score distribution directly from LRS queries.

If you can't rebuild the attempt from raw statements, the analytics you were sold live inside the vendor's walls and leave with the vendor if you switch. That's the whole point of xAPI — data you own — and it's exactly the part that gets skipped.

A procurement POC scenario you can hand to a vendor

Abstract tests get argued about. Concrete scenarios get executed. Give the vendor a defined proof-of-concept with pass/fail conditions, a timebox, and your own data.

Scenario: mid-size training team, roughly 1,800 learners across 40 courses, moving off a legacy platform.

1. 1
   Import a supplied QTI package of 200 items across 6 interaction types; produce a defect report showing exactly what did and didn't survive.
2. 2
   Configure LTI 1.3 against your sandbox LMS; demonstrate deep-linked resource creation, a completed attempt, and automatic grade pass-back.
3. 3
   Emit xAPI to a supplied LRS; deliver LRS queries that reproduce a cohort score distribution and one full learner journey.
4. 4
   Drop a learner from the LMS roster mid-POC and show the change reflected in the tool within the promised SLA.

Score it plainly: each deliverable is pass, partial, or fail, with the defect list attached. A well-defined two-week POC surfaces more real risk than three months of RFP responses, because the vendor has to touch your data instead of describing what their product does in the abstract. If you're already working from a structured procurement checklist for surge-ready assessment platforms, this POC becomes the evidence layer underneath those checkboxes.

A short real scenario

A corporate L&D team — roughly 2,000 employees, compliance-heavy, running annual recertifications — selected a platform largely on demo polish. The demo showed a clean LMS launch and a dashboard full of charts.

Post-signing, two things surfaced. First, the QTI import dropped competency tags on around 60% of their item bank, because the platform used its own taxonomy and silently ignored inbound metadata. Their "map results to competencies" reporting became impossible without re-tagging over a thousand items by hand — a job that ate somewhere in the range of six to eight weeks of a content specialist's time.

Second, xAPI statements came through with completed but no result.score, so the compliance team could prove someone finished a recert but not that they passed it. For an audit, "finished" is worthless.

None of this was hidden maliciously. It just never got tested against their data. When they ran a proper round-trip QTI test and an LRS reconstruction test on the next procurement cycle, they caught an equivalent metadata-loss issue in a finalist during the two-week POC — before signing — and used it to negotiate the vendor into building an import mapping layer as a contract condition. Same problem, caught on the cheap side of the signature.

Your vendor-acceptance checklist

Use this as the go/no-go gate.

1. 1
   [ ] QTI version and profile confirmed in writing (2.1 vs 3.0, supported interaction types)
2. 2
   [ ] Round-trip import/export tested with your own 200-item sample; defect report delivered
3. 3
   [ ] All your interaction types render and score correctly
4. 4
   [ ] Item metadata (competency, difficulty, accessibility) survives import
5. 5
   [ ] LTI 1.3 confirmed — no 1.1 fallback in launch payload
6. 6
   [ ] Deep Linking auto-creates line items
7. 7
   [ ] AGS grade pass-back verified with captured JSON payload
8. 8
   [ ] Grade updates on re-attempt instead of duplicating
9. 9
   [ ] NRPS roster sync tested with an add/drop, within stated SLA
10. 10
    [ ] xAPI statements captured raw and validated against the spec
11. 11
    [ ] Full attempt reconstructable from LRS statements alone
12. 12
    [ ] Consistent actor identity across all statements
13. 13
    [ ] Cohort-level report reproducible from LRS queries
14. 14
    [ ] Data-export and portability terms in the contract, not just the docs

Nothing signs until every relevant line has a captured artifact behind it, not a verbal yes.

When this level of rigor makes sense — and when it doesn't

When it's worth it: any migration involving an existing item bank you care about, any environment where grades must land in an LMS gradebook automatically, and any compliance context where you need defensible, exportable records. If your assessments feed audits, certifications, or hiring decisions, skipping the round-trip and LRS reconstruction tests is how you end up with data you can't trust.

When it's overkill: a small pilot with a handful of items you'd happily re-author, no LMS grade dependency, and no long-term reporting need. If you're spinning up a 30-question quiz for a one-off workshop, a full QTI round-trip and LRS reconstruction is more process than the decision deserves.

Who should slow down before doing this: teams without an architecture partner who can actually inspect a JWT, read an xAPI statement, and diff a QTI package. The tests only work if someone can execute them. If nobody on the team can capture and validate a payload, the acceptance criteria become another checkbox — which is exactly the problem you started with. In that case, bring in that skill for the POC window specifically. It's a short engagement with high leverage.

The systems view

Interoperability isn't three separate features — it's one connected flow. Content comes in through QTI, gets launched and graded through LTI, and generates the activity record through xAPI. A break at any point breaks the whole chain: lossy QTI import means your competency reporting is wrong no matter how good your LRS is; broken AGS means grades never reach the system of record; weak xAPI means you own the outcomes on paper but can't actually query them.

As programs scale, these breaks compound. At 200 learners you can patch a broken roster sync by hand. At 2,000 across dozens of courses, manual patching becomes a full-time role nobody budgeted for, and the small metadata losses you shrugged off in month one turn into reports leadership can't rely on. The failure points aren't dramatic — they're quiet, they accumulate, and they show up as trust erosion in the data long after the contract is signed.

Here's a simple workflow diagram showing QTI import → LTI launch/grade flow → xAPI statement generation and LRS reconstruction.

Run these acceptance tests before signing because the standards exist precisely to keep your content and your data portable. The moment you accept "compliant" without proof, you've handed that portability back to the vendor and turned an open standard into a lock-in.

Test the round-trip. Capture the payloads. Rebuild the attempt from the LRS. If the platform can survive that on your data, in your environment, with your team watching — you're buying interoperability instead of a slide about it.