Individual Project Guidelines (75%)
CUI6122: Successful Test Making for Teachers
Project Overview
This individual project requires you to develop a classroom test in your subject area or specific learning domain, administer it to students, analyze the results using Rasch measurement techniques, and provide evidence-based recommendations for improvement. The project emphasizes both the technical aspects oftest development and the practical application of measurement theory to enhance classroom assessment.
Your completed project should be approximately 2000 words in English (excluding appendices, references, and test items) and will demonstrate your ability to apply the principles learned in this course to real educational assessment contexts.
Project Structure
Your project should be organized into three major sections:
Part A: Test Development (25%; approx. 600 words)
1. Introduction
2. Table of Specifications (TOS)
3. Test Design
Part B: Test Administration and Analysis (35%; approx. 1000 words)
1. Data Collection
2. Rasch Analysis using ShinyTAM
Part C: Reflection and Recommendations (15%; approx. 400 words)
1. Interpretation of Results
2. Recommendations for Improvement
3. Conclusion
Sample Test Domains
You may choose any subject area relevant to your teaching experience or interests. Some examples include:
• Mathematics (e.g., algebra, geometry, statistics)
• Science (e.g., biology, chemistry, physics)
• Language Arts (e.g., reading comprehension, grammar)
• Social Studies (e.g., history, geography)
• Foreign Languages
• Computer Science
• Arts or Music Theory
Detailed Requirements
1. Test Development (25%; approx. 600 words)
1.1 Introduction
• Clearly state the purpose of your test and its role in the learning process
• Describe the specific learning domain being assessed
• Explain how the test aligns with curriculum standards or learning objectives
• Provide context for the test (e.g., formative/summative)
1.2 Table of Specifications
• Create a detailed TOS showing the relationship between:
o Content areas/topics
o Cognitive levels (using Bloom's Taxonomy)
o Number of items and marks allocated
• Explain your rationale for the distribution of items
• Ensure appropriate balance between knowledge recall and higher-order thinking
1.3 Test Design
• Develop 20-30 multiple-choice questions that:
o Follow best practices for item writing
o Have clear and concise stems
o Include one correct answer and three plausible distractors
o Vary in difficulty based on Bloom's Taxonomy levels
• Provide a clear marking scheme (A way for teachers to mark the tests efficiently)
• Organize items in a logical sequence (typically from easier to more difficult)
2. Test Administration and Analysis (35%; approx. 1000 words)
2.1 Data Collection
• Describe how the test was administered
• Provide information about the student sample (anonymized; use codenames)
• Explain data entry procedures (briefly mention how the test responses are placed on the Excel file: download here)
• Note any issues encountered during administration
2.2 Rasch Analysis
Using ShinyTAM (https://shinytam.eduhk.hk/en/), conduct the following analyses:
• Item Difficulty Analysis
o Report logit values for all items
o Identify the most difficult and easiest items
o Analyze whether difficulty distribution matches expectations
• Person Ability Analysis
o Report ability measures for the sample
o Analyze the distribution of student abilities
o Identify any unusual performance patterns
• Fit Statistics
o Identify items with problematic fit (Item fit values outside 0.5-1.5 range)
o Analyze potential reasons for misfit
o Evaluate person fit patterns
• Wright Map Interpretation
o Analyze the alignment between item difficulties and person abilities
o Identify gaps or clusters in the item difficulty distribution
o Evaluate the targeting of the test to the student population
• Reliability Analysis
o Report person and item reliability indicators
o Evaluate the overall test reliability
o Discuss separation indices
• Distractor Analysis
o Evaluate the functioning of response options
o Identify non-functioning distractors
o Analyze whether higher-ability students select correct answers more frequently
3. Reflection and Recommendations (15%; approx. 400 words)
3.1 Interpretation of Results (10%)
• Synthesize findings from the Rasch analysis
• Discuss whether the test achieved its intended purpose
• Evaluate whether item difficulties matched expectations
• Identify strengths and weaknesses in the test
3.2 Recommendations for Improvement (15%)
• Provide specific, evidence-based recommendations for:
o Revising problematic items (provide revised versions)
o Improving test structure
o Enhancing the match between test difficulty and student ability
• Connect findings to instructional implications
• Suggest how the test could better serve its educational purpose
3.3 Conclusion (5%)
• Summarize key findings and their significance
• Reflect on the test development and analysis process
• Discuss implications for your future assessment practices
• Connect project outcomes to broader assessment principles
Submission Requirements (Due: 12 April 2024)
Staggered Timeline:
• March 08: Test Development (Draft; Word file; Submit in Moodle)
• March 22: Rasch Data Analysis (Draft; Excel file; PDF file; Submit in Moodle)
• March 29: Preliminary Full Draft (Word file; Submit in Moodle)
• April 03: In-class Writing of Full Draft
• April 12: Submission of Final Paper (with attachments and appendices; in Moodle)
Your final submission should include:
1. Main Report (approximately 2000 words)
o Follow the structure outlined above
o Include relevant tables, figures, and visualizations from ShinyTAM outputs
2. Appendices
o Complete test items with answer key
o Table of Specifications
o Raw data (anonymized; Excel file)
o ShinyTAM output reports (PDF format)
o Any additional supporting materials
3. References
o APA 7th edition format
o If possible, include relevant literature on assessment principles, Rasch measurement, and subject-specific educational assessment