How Examulator & Exam Advantage Questions Are Generated And Vetted

I get asked questions about how our question pools are generated, their validity, etc., on a regular basis. For that reason, I have created this page which hopefully clears up any questions you may have. Vocal critics (usually competitors who charge for their question pools) levy accusations of "AI slop", "poor quality", "IP theft" and other claims. All I can say is this: I have passed 15 cybersecurity certification exams in the past 16 months, all on my first attempt, and for each exam I have used, at some level, these very pools as part of my study regime. This document refers to Examulator's pipeline specifically, however, Exam Advantage uses a similar pipeline for its content and question generation.

Overview

First, let me categorically state that Examulator's question pools do not contain any intellectual property (actual exam questions) from the certifying bodies. While I encourage those who have taken an exam to e-mail me and let me know the good and bad of Examulator's pools against their actual exam experience, under no circumstances do I ask anyone to violate their NDA and provide me with actual exam questions. Such actions would not only be illegal (IP theft) but would be unethical and a violation of the Code of Ethics of the bodies I am certified by.

Examulator's pools are not an "exam dump". Every Examulator question pool and Exam Advantage question is built from one source of truth, a centralized SQL Server database using a proprietary program, then published as a check-summed zip that the apps verify again before installing. Nothing reaches a user's device without passing three gates: checks while the pool is built, a checksum match when it is downloaded, and a structural validation before it is installed.

The pipeline has five stages:

Source of truth: the SQL Server database

All questions live in one SQL Server database, organized as agency → exam → domain → objective → question → answers. Both generations share the exam structure; each generation has its own question and answer tables. A summary of the tables are:

AGENCIES: information on certifying bodies, e.g. ISC2, ISACA, CompTIA, EC-Council

EXAMS: one row per exam edition 

DOMAINS: the exam's domains, with their published weightings

OBJECTIVES: the objectives under each domain; spans pool generations, so older pools leave many objectives empty

QUESTIONS_V2 / ANSWERS_V2 Gen 2 question text, one explanation per question

QUESTIONS_V3 / ANSWERS_V3 Gen 3 question text and a key reference; an explanation on every answer; IRT columns

Every question, answer, objective, domain and exam carries an archive flag. Retired content is archived rather than deleted, so results that refer to it stay readable.

Writing and auditing questions

Canon objectives first

Every question hangs off an objective, and every objective belongs to a domain of one exam.

When a question pool is built, the first step is populating the AGENCIES, EXAMS, and DOMAINS tables based on the certifying body's exam curriculum. Then, the syllabus published by the certifying body for that specific exam is converted into what we refer to as "canon objectives" and are populated in the OBJECTIVES table.

The OBJECTIVES table contains an objective number which is composed of two parts, separated by a space. The first part carries each exam's published outline in the agency's own numbering system (for example 1.1 or 1A1). This is referred to as a "canon objective". Canon objectives map directly to the exam syllabus published in exam outlines, blueprints, etc. of the respective certifying body (each body uses different terminology to refer to their exam's objectives, for purposes of this web page we'll use the generic term "syllabus").

Following the canon objective -- the second part of the objective number, after the space -- sits Examulator's own finer numbering system (for example a.b.c), which lets generation and reporting spread questions evenly across a syllabus instead of letting a few popular topics dominate. OBJECTIVES spans pool generations, so many fine-grained objectives have no questions yet in the older pools. That is expected: they fill as later pool generations are written.

Canon objective numbers follow whatever convention (as best as possible) the certifying body uses. They can be numeric, alphabetic, or a combination. Examulator's numbering system is always hierarchical with alphabetic, never numeric, indicators.

Examulator Subject Generation

Certification Tool and Die maintains an extensive online reference library of over 2,500 cybersecurity-related reference books which have been OCRd and subject indexed by topic. In addition, we are a member of Anthropic's Cyber Verification Program, with access to Anthropic's Mythos cybersecurity research model.

Using these sources, for each certification exam, we ask Claude to construct a list of subjects for each canon objective which an individual studying for that specific exam would reasonably be expected to have knowledge about, and to group the subjects accordingly into the appropriate hierarchical structure. We do this using an exam-specific prompt which is individually tuned in scope and voice for that exam. The output of these requests become the "a.b.c" component under each certifying body canon objective. We refer to this as the "Examulator Subject"; it's objective numbering has the format "1.2 a.b", "1A1 a.b", as mentioned earlier.

During the generation process we also ask for an appropriate level of granularity so students can hone in on specific areas where their knowledge is lacking. This is why some subjects have a single level (e.g. a) while others may have 2 or more (e.g. a.b.c) below the canon objective. Some subjects (for example, the NIST Cybersecurity Framework) require a deeper level of granularity for some exams than other subjects or even the same topic in a different exam.

The results of these generation tasks are then populated into the OBJECTIVES table, and used in the next step for question generation.

Question & Answer Generation

The next step in the process involves generating the question bank for each individual subject. This is accomplished using a custom-built prompt which takes into account the subject matter, the reference material available on the topic (such as NIST or ISO documents), and the perspective of the exam taker (e.g. cybersecurity analyst, auditor, information security manager, etc.). For each subject, a custom prompt is generated and fed into a Claude model asking for questions that meet the requirements. The choice of model and effort level depends on the examination and subject. The results include:

We then determine the optimal pool size. The rule of thumb is you need 4 to 6 times the length of a single operational exam or 8 to 12x the length of an adaptive test per domain/subtest. As ISC2 CAT exams are 100 questions, the top end would be 1200 questions, linear exams @ 125 questions would require 750 questions, and other exams like ISACA exams at 150 questions/exam would require 900 questions. We then add 2 orders of magnitude to those numbers to get the total pool size.

That pool size is then divided by the domain weights so the pools are properly balanced. For example, a question pool of 120,000 questions where a domain (ex. Domain 1) is weighted 10% would get 12,000 questions (120,000 x .10). The domain count is divided by the number of subjects in that domain (for example, say there are 100 subjects in Domain 1), or in the example, each subject would get 120 questions built for that specific subject (12,000 ÷ 100). The minimum number of questions any subject will get is 10, regardless of calculated number.

The typical Examulator question pool averages about 200,000 questions.

Triage and Remediation

During generation two dozen quality checks are performed against each individual question, answer, explanation and key reference. Questions that fail a check are pulled aside into holding tables for rework rather than being edited on the live tables, and objective-mapping problems are fixed in dedicated remediation passes before a pool is rebuilt. Some of these questions, if repairable, ultimately become "unscored" questions. Others are discarded.

Question Pool Audit

For the questions that pass the quality check gate, before a question is designated as releasable to a public question pool, it goes through an automated content audit using a second OpenAI model, one item at a time:

What the audit looks for: one defensible key; distractors that are plausible but clearly wrong; no answer that gives itself away by length or wording; a stem that actually tests its mapped objective; explanations consistent with the key; current terminology and standards and hallucinations.

These results are then fed back into Claude for integration with the original question results.

When there is disagreement: When there is substantial disagreement on the verdict reasoning or other factors, the questions are fed into a third model (DeepMind, DeepSeek, etc.) as a tie-breaker.

Calibration: item parameters and scoring

Examulator scores with a three-parameter logistic (3PL) item response theory model. 

Stored parameters

Each question carries three raw byte values from 1 to 254 (0 is reserved, see Domain 0 below), converted at run time:

difficulty → b → ability at which the item is 50/50 (above guessing); (raw − 128) × 6 / 253; about −3.0 to +3.0

discrimination → a → how sharply the item separates weaker from stronger candidates; (raw − 1) / 253 × 2.5; 0 to 2.5

guessing → c → chance a candidate with no knowledge gets it right; (raw − 1) / 253 × 0.35; 0 to 0.35

The 1–254 scale is used to reduce the size of the pools, rather than storing data as a decimal number, it is stored as a tinyint and then converted into the proper value at runtime. difficulty is also banded (Easy below 64, Moderate 64–127, harder above).

Where the numbers come from

Domain 0 (beta items). Domain 0 holds unscored beta questions, as on the real exams. They have difficulty, discrimination and guessing all 0, appear in tests, and never count toward the score.

Scoring a test

Adaptive versus linear. CISSP, CCSP, SSCP and CC run as computerized adaptive tests: the next item is the most informative one at the current θ; from the minimum length (100 items) on, the test stops as soon as the 95% confidence interval (θ ± 1.96 × SE) lies wholly above or below the cut; otherwise it runs to the maximum length and passes if θ ≥ 0. A CAT score is θ on the agency scale. The other exams, and quizzes, are linear: a linear score is a difficulty-weighted ratio (sum of difficulty over correct scored items ÷ sum over all scored items), a quiz score is the plain fraction correct, and both ratios are mapped through the agency's scoring profile (ISC2 0–1000 with 700 to pass, a straight ×1000; ISACA 200–800 with 450 to pass). Beta items are left out of every score.

Packaging and publishing

An automated script builds and publishes the question pools to our Github repository. Each publication run does five things:

Vetting on the device

The apps do not trust a pool because it arrived. Every pool passes the same gates whether it was downloaded, imported or dropped into a folder.

Finding updates. The apps look in order at the GitHub releases of the pool repository for their generation, then the Drive folder (older versions), then the versions page (even older versions). The update offer compares the zip's full YYDDD.HHmm with the upload recorded in ExamData/PoolVersions.json. Without a record (a pool from the drop folder, an import, an older install) it compares the build day YYDDD with the installed pool's PRAGMA user_version.

Gate 1: integrity of the download. A download must match a published hash or it is refused: the SHA-256 from GitHub's asset digest or the release's SHA256SUMS.txt (or the versions page), or Drive's MD5 for a Drive copy.

Gate 2: is it a pool at all? The file must begin with a valid SQLite header.

Gate 3:  a pool validation routine is run on the unpacked file before it replaces anything:

Reading. Format-2 documents are decompressed with the pool's own dictionary; readers are cached by pool path plus the dictionary's SHA-256, so a replaced pool can never be read with a stale dictionary.

The PC drop folder. The PC program folder's wwwroot\data is checked on every start: new or newer pools are validated and installed, a refusal is reported once, and nothing there is ever deleted.

Tests. The build pipeline cover these paths (format 2 reading, generation detection, update checks) and must both pass before a release.

Known limits

*Unlike other sites on the internet offering question pools, Examulator is designed to work offline. It does not collect your data to share with us. We do not require an account, there is no telemetry sent back to us from the software. Since Examulator is not an online service, we cannot dynamically use your test results to tune the IRT parameters on questions. Users can contribute their exam results anonymously to us so we may adjust the values, but this feature is entirely opt-in, by default as the product is shipped to the user the functionality is disabled, and enabling it requires specific consent of the user.