- Job Title: Psychometrician
- Work Unit: System Excellence Division, Analytics Branch
- Ministry: Alberta Education
- Competition Number: 1055429
Reporting to the Senior Manager, Scoring and Reporting, the Psychometrician is one of several professional education specialists collaborating and providing sophisticated psychometric and IT leadership and advice regarding the design, development, and enhancement of all testing, marking, scoring, test administration, accountability, and results reporting materials and processes.
- Co-leads the planning and development of business systems, system development teams;
- Manages the psychometric aspects of Division systems and functions; and
- Leads and coordinates the operations-based psychometric consultative, research, and communication functions for other education professionals.
Responsibilities and Activities
Alberta Test Design, Development and Standards Incorporate Sound Psychometric Principles and Procedures.
- Initiates, manages, and coordinates projects to develop, enhance, or maintain test design and development processes that serve the psychometrically driven business needs of the branches or units responsible for development, document production, and/or test administration.
- Collaborates with and provide guidance to Exam Team Leaders in test design and development to ensure that tests follow sound psychometric principles and procedures.
- Designs and leads stakeholders through standard setting and confirmation sessions to ensure that appropriate scoring and reporting standards are being used.
- Provides expert advice and conducts standard setting for new courses and new curricula within existing courses.
- Collaborates with other psychometricians to ensure that proper psychometric standards have been applied to field test items and final examination forms.
- Promotes quality assurance and continuous improvement by collaborating with members of the branches or units responsible for development, document production, and test administration, as well as with other psychometricians and members of the Online/IT Services Unit.
Business Rules for Scoring & Reporting Conform to Sound Psychometric and Information Technology Parameters.
- Collaborates with Assessment Directors and Exam Team Leaders to develop sound, consistent, and usable statements of business rules across units for use in business systems development.
- Leads project teams of Division staff and/or contract staff to ensure that systems will meet Division and Branch business needs and follow Division business rules.
- Provides expert advice and conducts standards setting for new courses and new course curricula.
- Designs, develops, manages, and monitors psychometric and statistical procedures for all Alberta provincial examinations and tests.
- Manages the processes of defining and designing psychometric aspects of Assessment systems and functions.
- Designs and monitors the production of detailed and multiyear summary reports of test results at different levels (e.g., students, schools, jurisdictions, and the province) to provide useful, accessible and accurate information.
- Reviews and provides feedback about test administration, scoring, and marking designs, processes, statistics, and materials to help ensure that they conform to the psychometric and information technology parameters established by the Division for that course and testing program.
- Provides leadership and psychometric support in proper interpretation of the results and standards for provincial assessments.
- Maintains a comprehensive understanding of Assessment data held in Ministry files to effectively deal with stakeholder information requirements and requests.
- Provides leadership and service related to data analysis and information requests.
- Conceives, designs, and produces secondary analysis reports of achievement tests, diploma examinations, student learning assessments, and national and international assessment data.
- Maximizes scoring validity and accuracy when presented with unique situations and events.
Psychometric Research Provides Leadership and Value-Added Support for Ministry Strategic, Operational, and Budget Planning.
- Designs, develops, implements, and leads operations-based psychometric research and keeps abreast of assessment related issues and trends in support of Alberta assessment programs.
- Designs, plans, manages, and coordinates cutting edge initiatives such as online assessment methods, statistical equating, computer based/adaptive testing, and automatic essay scoring.
- Conceives, designs, and conducts policy-based research studies to inform Division and Branch decision-making.
- Provides leadership by compiling, reviewing, summarizing, and interpreting relevant psychometric and educational assessment reports and literature to identify emerging issues of relevance to Alberta.
- Meets and consults with psychometric and educational assessment academics and professionals to gain advice in support of Division needs and priorities.
- Identifies through research and development activity findings assessment and information technology related issues and trends that have strategic, operational, and budget implications for the Division and Branches
- Prepares briefing notes and recommendations in relation to identified issues and trends.
- Collaborates to ensure that assessment related legislation, regulations, policies, ministerial orders, and practices are developed with full consideration of current and emerging psychometric and educational assessment trends and technologies.
- Initiates and provides ad hoc reporting services as well as information requests from Division and Branch staff, and from stakeholders and members of the public.
- Initiates and conducts proposals for changes which improve the psychometric quality of the tests and systems.
- Serves as the psychometric expert on all vendor-developed psychometrically-based IT products.
- Contributes to the development of psychometric expertise through mentoring and the design and delivery of formal training sessions.
The Psychometrician is one of several positions charged with designing, developing, managing, and monitoring psychometric and statistical procedures for all Alberta provincial examinations and tests.
These positions have a great effect on policy change at the executive level by providing advice, analyses and data based evaluations of new initiatives. As well they determine the psychometric consequences of the change, design, develop, and test changes to scoring and reporting systems resulting from the change, work with systems development staff to implement the changes in scoring and reporting systems, and communicate to Exam Team Leaders, Directors, and other management staff the consequences of the change, its effects on comparability of results over time, and any opportunities the change may create.
With the implementation of a new exam, or after a change in curriculum, the cut scores for provincial standards have to be set based on a proven standard setting process. The Psychometrician guides the creation of the documentation and materials to be used for the standard setting by managing and overseeing the session, guiding the Exam Team Leaders, Examiners, and subject matter experts through the procedure, process the data collected, provide and explain item/test-level statistical information, and provide procedural and consequence feedback to the participants to ensure that the obtained cut-scores are valid, fair, and defensible.
Knowledge, Skills and Abilities
- PhD or equivalent in educational measurement or a related field, and experience in statistics with proven ability to solve psychometric and statistical problems.
- Expert knowledge of the mathematics behind psychometric statistics used to build and test custom scoring and reporting systems.
- Working knowledge of databases for the psychometric aspects of Division systems and functions using software applications such as SPSS, EQS, SIBTEST, Iteman, Lertaps, BILOG, CIPE, PARSCALE, TESTGRAF, Visual Basic, MS Access, MS Excel.
- Working knowledge of the education system, structures, and policies of school authorities.
- Working knowledge of the applicable legislation, regulations, policies and protocol governing Assessment work, including the Ministerial Order on Student Learning (#0010/2013), Curriculum Redesign Project, Programs of Study for all subjects, Guide to Education K-12, and the School Act.
- Working knowledge of systems development methodologies and practices and the ability to work with systems analysts to build and test scoring and reporting systems.
- Thorough knowledge of Corporate Information and Communications Technology Business Plan as well as Division processes and interrelationships used to design psychometrically sound systems that apply Division business rules and practices.
- Expert understanding of Classical Test Theory (CTT) and Item Response Theory (IRT) and the ability to use that information to design scoring and reporting systems and to provide expert advice and support to others in the Division and Branches.
- Expert understanding of learning theory and cognition as it applies to large-scale assessment and the ability to use that information when designing testing materials, scoring and reporting systems, and assisting others to use and interpret test scores in ways that are meaningful and appropriate.
- Expert knowledge and ability to apply methods of data visualization and applied statistical modeling.
- Expert understanding of the Standards for Educational and Psychological Testing (1999) and the Principles for Fair Student Assessment Practices for Education in Canada (1993) and the ability to evaluate and interpret Division business practices in relation to those standards.
- Expert understanding of national and international assessment issues and trends and the ability to use that information to help shape Division and Branch policy and practice.
- Expert knowledge and ability to design and conduct policy based psychometric research studies and to understand and interpret the significance of research studies for assessment practice in Alberta, including expert knowledge of statistical analysis techniques such as univariate and multivariate analysis of variance, linear and nonlinear regression techniques, factor analysis, scaling methods, equating, parametric and non-parametric techniques, and the ability to use standard and custom data analysis software to apply these techniques correctly and appropriately.
Significant contact with Exam Team Leaders, Directors and other Psychometricians on a daily basis for discussion, information exchange, cross branch achievement issues, discussions on programs, tests and electronic scoring, and to provide expertise in test development, design and scoring.
Contact with representatives from national/international assessment initiatives, academics from post-secondary institutions, Auditors, Superintendents, Principals, Teachers, Parents, Exam Team Leaders, and the Directors of development and administration branches in Assessment on an ongoing basis while designing, developing, and implementing business procedures with psychometric implications.
Input of many stakeholders including other branches and units in Assessment (e.g., Provincial Achievement Testing and Document Production, Diploma Examinations, and Test Administration) and other Alberta Education and Government Departments (e.g., ADM office, Auditor, and Deputy Minister/Minister’s Office) is sought on a regular basis when designing, developing, and implementing business procedures with psychometric implications.
Psychometricians are required to find solutions that fit across all stakeholder groups.
Though no formal supervisory role, the Psychometricians have a significant leadership/consultative role with Exam Team Leaders and stakeholders. In addition, as a required responsibility, positions use personal management skills and psychometric expertise to train new Exam Team Leaders, new Psychometricians, contract data analysis staff and support staff. The positions are relied upon for their specialist knowledge in test design, development, administration, scoring and reporting activities in exam and diploma management.
The Government of Alberta is committed to a diverse and inclusive public service that reflects the population we serve to best meet the needs of Albertans. Consider joining a team where diversity, inclusion and innovation are valued and supported. For more information on diversity and inclusion, please visit the Diversity and Inclusion Policy.