How Examulator & Exam Advantage Questions Are Generated And Vetted
I get asked questions about how our question pools are generated, their validity, etc., on a regular basis. For that reason, I have created this page which hopefully clears up any questions you may have. Vocal critics (usually competitors who charge for their question pools) levy accusations of "AI slop", "poor quality", "IP theft" and other claims. All I can say is this: I have passed 15 cybersecurity certification exams in the past 16 months, all on my first attempt, and for each exam I have used, at some level, these very pools as part of my study regime. This document refers to Examulator's pipeline specifically, however, Exam Advantage uses a similar pipeline for its content and question generation.
Overview
First, let me categorically state that Examulator's question pools do not contain any intellectual property (actual exam questions) from the certifying bodies. While I encourage those who have taken an exam to e-mail me and let me know the good and bad of Examulator's pools against their actual exam experience, under no circumstances do I ask anyone to violate their NDA and provide me with actual exam questions. Such actions would not only be illegal (IP theft) but would be unethical and a violation of the Code of Ethics of the bodies I am certified by.
Examulator's pools are not an "exam dump". Every Examulator question pool and Exam Advantage question is built from one source of truth, a centralized SQL Server database using a proprietary program, then published as a check-summed zip that the apps verify again before installing. Nothing reaches a user's device without passing three gates: checks while the pool is built, a checksum match when it is downloaded, and a structural validation before it is installed.
The pipeline has five stages:
Author and audit questions in the database against each exam's objectives taxonomy, with AI-assisted generation and audit passes followed by human review.
Calibrate each question with Item Response Theory (IRT) parameters, or mark it as an unscored beta question.
Build one SQLite pool per exam edition (Gen 2 and Gen 3 separately), compressed and indexed, failing on inconsistent data.
Publish only the pools whose content changed, as zips with SHA-256 checksums, to GitHub releases (and Google Drive, deprecated).
Verify on the device: the app refuses any download that doesn't match its published checksum, any pool for the other generation, and any file that fails validation.
Source of truth: the SQL Server database
All questions live in one SQL Server database, organized as agency → exam → domain → objective → question → answers. Both generations share the exam structure; each generation has its own question and answer tables. A summary of the tables are:
AGENCIES: information on certifying bodies, e.g. ISC2, ISACA, CompTIA, EC-Council
EXAMS: one row per exam edition
DOMAINS: the exam's domains, with their published weightings
OBJECTIVES: the objectives under each domain; spans pool generations, so older pools leave many objectives empty
QUESTIONS_V2 / ANSWERS_V2 Gen 2 question text, one explanation per question
QUESTIONS_V3 / ANSWERS_V3 Gen 3 question text and a key reference; an explanation on every answer; IRT columns
Every question, answer, objective, domain and exam carries an archive flag. Retired content is archived rather than deleted, so results that refer to it stay readable.
Writing and auditing questions
Canon objectives first
Every question hangs off an objective, and every objective belongs to a domain of one exam.
When a question pool is built, the first step is populating the AGENCIES, EXAMS, and DOMAINS tables based on the certifying body's exam curriculum. Then, the syllabus published by the certifying body for that specific exam is converted into what we refer to as "canon objectives" and are populated in the OBJECTIVES table.
The OBJECTIVES table contains an objective number which is composed of two parts, separated by a space. The first part carries each exam's published outline in the agency's own numbering system (for example 1.1 or 1A1). This is referred to as a "canon objective". Canon objectives map directly to the exam syllabus published in exam outlines, blueprints, etc. of the respective certifying body (each body uses different terminology to refer to their exam's objectives, for purposes of this web page we'll use the generic term "syllabus").
Following the canon objective -- the second part of the objective number, after the space -- sits Examulator's own finer numbering system (for example a.b.c), which lets generation and reporting spread questions evenly across a syllabus instead of letting a few popular topics dominate. OBJECTIVES spans pool generations, so many fine-grained objectives have no questions yet in the older pools. That is expected: they fill as later pool generations are written.
Canon objective numbers follow whatever convention (as best as possible) the certifying body uses. They can be numeric, alphabetic, or a combination. Examulator's numbering system is always hierarchical with alphabetic, never numeric, indicators.
Examulator Subject Generation
Certification Tool and Die maintains an extensive online reference library of over 2,500 cybersecurity-related reference books which have been OCRd and subject indexed by topic. In addition, we are a member of Anthropic's Cyber Verification Program, with access to Anthropic's Mythos cybersecurity research model.
Using these sources, for each certification exam, we ask Claude to construct a list of subjects for each canon objective which an individual studying for that specific exam would reasonably be expected to have knowledge about, and to group the subjects accordingly into the appropriate hierarchical structure. We do this using an exam-specific prompt which is individually tuned in scope and voice for that exam. The output of these requests become the "a.b.c" component under each certifying body canon objective. We refer to this as the "Examulator Subject"; it's objective numbering has the format "1.2 a.b", "1A1 a.b", as mentioned earlier.
During the generation process we also ask for an appropriate level of granularity so students can hone in on specific areas where their knowledge is lacking. This is why some subjects have a single level (e.g. a) while others may have 2 or more (e.g. a.b.c) below the canon objective. Some subjects (for example, the NIST Cybersecurity Framework) require a deeper level of granularity for some exams than other subjects or even the same topic in a different exam.
The results of these generation tasks are then populated into the OBJECTIVES table, and used in the next step for question generation.
Question & Answer Generation
The next step in the process involves generating the question bank for each individual subject. This is accomplished using a custom-built prompt which takes into account the subject matter, the reference material available on the topic (such as NIST or ISO documents), and the perspective of the exam taker (e.g. cybersecurity analyst, auditor, information security manager, etc.). For each subject, a custom prompt is generated and fed into a Claude model asking for questions that meet the requirements. The choice of model and effort level depends on the examination and subject. The results include:
Gen 2 (deprecated): a stem, a set of answers, one correct answer and one explanation for the whole question
Gen 3 (current): a stem, a key reference (the standard, framework or body-of-knowledge section the answer rests on) and an explanation on every answer, so a candidate learns why each distractor is wrong and not only why the key is right.
We then determine the optimal pool size. The rule of thumb is you need 4 to 6 times the length of a single operational exam or 8 to 12x the length of an adaptive test per domain/subtest. As ISC2 CAT exams are 100 questions, the top end would be 1200 questions, linear exams @ 125 questions would require 750 questions, and other exams like ISACA exams at 150 questions/exam would require 900 questions. We then add 2 orders of magnitude to those numbers to get the total pool size.
That pool size is then divided by the domain weights so the pools are properly balanced. For example, a question pool of 120,000 questions where a domain (ex. Domain 1) is weighted 10% would get 12,000 questions (120,000 x .10). The domain count is divided by the number of subjects in that domain (for example, say there are 100 subjects in Domain 1), or in the example, each subject would get 120 questions built for that specific subject (12,000 ÷ 100). The minimum number of questions any subject will get is 10, regardless of calculated number.
The typical Examulator question pool averages about 200,000 questions.
Triage and Remediation
During generation two dozen quality checks are performed against each individual question, answer, explanation and key reference. Questions that fail a check are pulled aside into holding tables for rework rather than being edited on the live tables, and objective-mapping problems are fixed in dedicated remediation passes before a pool is rebuilt. Some of these questions, if repairable, ultimately become "unscored" questions. Others are discarded.
Question Pool Audit
For the questions that pass the quality check gate, before a question is designated as releasable to a public question pool, it goes through an automated content audit using a second OpenAI model, one item at a time:
The auditor model gets the stem, all answers, the marked key and the explanation(s).
It returns a verdict, KEEP, MODIFY (with the proposed fix) or REPLACE, plus its reasoning.
MODIFY and REPLACE go back to the holding tables for a reviewed fix; KEEP passes.
What the audit looks for: one defensible key; distractors that are plausible but clearly wrong; no answer that gives itself away by length or wording; a stem that actually tests its mapped objective; explanations consistent with the key; current terminology and standards and hallucinations.
These results are then fed back into Claude for integration with the original question results.
When there is disagreement: When there is substantial disagreement on the verdict reasoning or other factors, the questions are fed into a third model (DeepMind, DeepSeek, etc.) as a tie-breaker.
Calibration: item parameters and scoring
Examulator scores with a three-parameter logistic (3PL) item response theory model.
Stored parameters
Each question carries three raw byte values from 1 to 254 (0 is reserved, see Domain 0 below), converted at run time:
difficulty → b → ability at which the item is 50/50 (above guessing); (raw − 128) × 6 / 253; about −3.0 to +3.0
discrimination → a → how sharply the item separates weaker from stronger candidates; (raw − 1) / 253 × 2.5; 0 to 2.5
guessing → c → chance a candidate with no knowledge gets it right; (raw − 1) / 253 × 0.35; 0 to 0.35
The 1–254 scale is used to reduce the size of the pools, rather than storing data as a decimal number, it is stored as a tinyint and then converted into the proper value at runtime. difficulty is also banded (Easy below 64, Moderate 64–127, harder above).
Where the numbers come from
Gen 2 pools are deliberately 1PL: only difficulty varies, and discrimination and guessing are fixed at 127 (a ≈ 1.25, c ≈ 0.17) for every scored question.
Gen 3 pools carry a full 3PL triple per question. With no large body of real candidate responses yet, the triples are seeded: During question generation the model is asked to estimate difficulty and discrimination from the content. A weak discriminator gets c ≈ 0.21 and the sharpest gets c ≈ 0.05. When there is disagreement on these figures during the audit phase, the third model acts as a tie-breaker.
Domain 0 (beta items). Domain 0 holds unscored beta questions, as on the real exams. They have difficulty, discrimination and guessing all 0, appear in tests, and never count toward the score.
Scoring a test
For each scored response, the probability of a correct answer at ability θ is P(θ) = c + (1 − c) / (1 + e^(−a(θ − b))).
Ability θ is the maximum-likelihood estimate, found by damped Fisher scoring and clamped to ±3.5 (an all-right or all-wrong test has no finite MLE).
Precision comes from the Fisher information: each item adds I(θ) = a² (P − c)² (1 − P) / ((1 − c)² P), the test information TI is their sum, and the standard error is SE = 1 / √TI.
The pass point is θ = 0. On the ISC2 scale an adaptive result is 700 + 85.7 × θ (300 points per 3.5 θ units), so the cut is 700.
Adaptive versus linear. CISSP, CCSP, SSCP and CC run as computerized adaptive tests: the next item is the most informative one at the current θ; from the minimum length (100 items) on, the test stops as soon as the 95% confidence interval (θ ± 1.96 × SE) lies wholly above or below the cut; otherwise it runs to the maximum length and passes if θ ≥ 0. A CAT score is θ on the agency scale. The other exams, and quizzes, are linear: a linear score is a difficulty-weighted ratio (sum of difficulty over correct scored items ÷ sum over all scored items), a quiz score is the plain fraction correct, and both ratios are mapped through the agency's scoring profile (ISC2 0–1000 with 700 to pass, a straight ×1000; ISACA 200–800 with 450 to pass). Beta items are left out of every score.
Packaging and publishing
An automated script builds and publishes the question pools to our Github repository. Each publication run does five things:
Select. For every pool in the generation's folder it reads the build day (PRAGMA user_version), the question count and the file's SHA-256. Pools with no questions (stubs) are never published. A pool is published to a destination only if its SHA-256 differs from the one recorded in published.json for that destination, so an unchanged rebuild uploads nothing.
Zip. Each selected pool goes into Examulator-{KEY}-v{N}-{YYDDD}.{HHmm}-db.zip together with Exams.db. YYDDD is the pool's own build day (the number the apps compare with user_version), and HHmm is the time of this run, so two publishes on one day still order correctly.
Checksums. SHA256SUMS.txt lists exactly the zips made in this run.
Drive (deprecated). rclone copies the zips to Examulator/Question Pools/Generation {N} on Google Drive. Pools still go to Drive (only program zips stopped, after Drive's false malware flags). [Effective 11/5/2026 we no longer distribute new iterations of the pools or program files on Google Drive.]
GitHub. A release tagged with this run's YYDDD.HHmm in ctdltd/Examulator-Pools-v2 or -v3, carrying the zips and SHA256SUMS.txt, plus a changelog entry at the top of that repository's README. The apps read every release, so a pool left out of this run stays available from its earlier release.
Vetting on the device
The apps do not trust a pool because it arrived. Every pool passes the same gates whether it was downloaded, imported or dropped into a folder.
Finding updates. The apps look in order at the GitHub releases of the pool repository for their generation, then the Drive folder (older versions), then the versions page (even older versions). The update offer compares the zip's full YYDDD.HHmm with the upload recorded in ExamData/PoolVersions.json. Without a record (a pool from the drop folder, an import, an older install) it compares the build day YYDDD with the installed pool's PRAGMA user_version.
Gate 1: integrity of the download. A download must match a published hash or it is refused: the SHA-256 from GitHub's asset digest or the release's SHA256SUMS.txt (or the versions page), or Drive's MD5 for a Drive copy.
Gate 2: is it a pool at all? The file must begin with a valid SQLite header.
Gate 3: a pool validation routine is run on the unpacked file before it replaces anything:
the tables Exams, Domains, Objectives and Questions exist and Exams has exactly one row, otherwise "not an Examulator question pool";
the storage format is one this build understands, otherwise the user is told to update Examulator first;
the pool is for this generation (judged from the columns in format 1 or from the content documents in format 2: a k key or per-answer x means Gen 3, an e explanation means Gen 2). A pool for the other generation is refused, so it can never be opened by mistake;
PRAGMA quick_check returns ok, otherwise the file is reported damaged and the user is asked to download it again.
Reading. Format-2 documents are decompressed with the pool's own dictionary; readers are cached by pool path plus the dictionary's SHA-256, so a replaced pool can never be read with a stale dictionary.
The PC drop folder. The PC program folder's wwwroot\data is checked on every start: new or newer pools are validated and installed, a refusal is reported once, and nothing there is ever deleted.
Tests. The build pipeline cover these paths (format 2 reading, generation detection, update checks) and must both pass before a release.
Known limits
Parameters are estimates, not calibrations. Gen 3's difficulty and discrimination are seeded by a language model from the question text, and guessing is derived from discrimination by formula. Proper calibration needs real candidate responses fitted per item. Until that data exists*, the ranges (maximum discrimination 2.5, maximum guessing 0.35) are working placeholders.
*Unlike other sites on the internet offering question pools, Examulator is designed to work offline. It does not collect your data to share with us. We do not require an account, there is no telemetry sent back to us from the software. Since Examulator is not an online service, we cannot dynamically use your test results to tune the IRT parameters on questions. Users can contribute their exam results anonymously to us so we may adjust the values, but this feature is entirely opt-in, by default as the product is shipped to the user the functionality is disabled, and enabling it requires specific consent of the user.
Gen 2 is 1PL by design. Only difficulty varies. Its scores rank candidates sensibly but cannot reflect items that discriminate better or worse.
Not the real exam's CAT. The adaptive engine follows the published shape of ISC2's CAT (minimum and maximum length, a confidence-interval stop around the cut, beta items), but the item bank, the parameters and the scale mapping are Examulator's own. A score predicts readiness; it is not an equivalent of an official result. ISACA does not publish how raw performance maps onto its 200–800 scale, so the ISACA profile is an acknowledged approximation: anchored so the pass standard lands on 450, with the rest of the range filled piecewise.
Audits are only as good as the auditor. The audit catches most defects but not all, and a weak auditor model passes everything. Disputed or MODIFY items still need a human decision.
Fine-grained objectives are partly empty. Many of Examulator's sub-objectives have no questions in the current pools. They are meant to fill with later pool generations (Pool Generation 5).
Build checks are structural. The pool build pipeline proves that a pool is well formed, complete and readable. They do not prove that a question is correct. Correctness rests on the authoring and audit steps above.
Use of Translation features. When Translation (beta) is turned on, questions and explanations are translated on the fly by a third-party machine-translation service (Apple, Google or Microsoft). Translations are machine-generated and not reviewed; the English original is authoritative. We do not know how much AI, if any, is used by those translation services.
It's AI. AI can make mistakes. We readily acknowledge, unlike other sources of question pools, that we use AI extensively to craft our question pools. The entire build pipeline has been extensively crafted with Claude, and relies on Anthropic's and other frontier models to generate. This pipeline has been undergoing an continuous refinement process for over a year, since this project started with the "Generation 1" pools. In the spirit of continuous monitoring, we use lessons learned from each generation's pool builds to tweak and finely tune the process for the next generation. Despite our best efforts, AI can and does make mistakes. There are most certainly errors in the pools. Therefore, standard disclaimers apply (suitability of use, no guarantees, they're worth what you paid for, etc.)