Pre- and Post-Conference Workshops 2017

Analyzing PIAAC data with structural equation modeling in Mplus
Dozenten: Dr. Ronny Scherer (CEMO, Oslo); FDZ PIAAC (GESIS Mannheim) Datum: 03. bis 04. April 2017 Ort: GESIS Mannheim Abstract: Structural equation modeling (SEM) has become one of the most commonly applied statistical approaches to disentangle the relationships among latent variables across groups, over time, and at different analytical levels. The potential of SEM has been recognized in many areas, including educational sciences, sociology, psychology, and business. This workshop provides an introduction to the principles and procedures of basic and more advanced SEM in the context of international large-scale assessments such as PIAAC. Specifically, the following topics were covered: (a) Principles of latent variable modeling, (b) Model identification and specification, (c) Measurement models (including confirmatory factor analysis), (d) Parceling, (e) Structural regression models (including exploratory SEM), (f) Multi-group SEM (including measurement invariance testing), and (g) Indirect effects and moderation models. The workshop comprises lectures and practical sessions, in which participants put into practice basic and more advanced SEM with PIAAC data. Participants primarily used the statistical software Mplus; yet, code and syntax for AMOS and R (lavaan) were provided.

Analyzing PIAAC data with structural equation modeling in Mplus

Dozenten: Dr. Ronny Scherer (CEMO, Oslo); FDZ PIAAC (GESIS Mannheim)

Datum: 03. bis 04. April 2017

Ort: GESIS Mannheim

Abstract: Structural equation modeling (SEM) has become one of the most commonly applied statistical approaches to disentangle the relationships among latent variables across groups, over time, and at different analytical levels. The potential of SEM has been recognized in many areas, including educational sciences, sociology, psychology, and business.

This workshop provides an introduction to the principles and procedures of basic and more advanced SEM in the context of international large-scale assessments such as PIAAC. Specifically, the following topics were covered: (a) Principles of latent variable modeling, (b) Model identification and specification, (c) Measurement models (including confirmatory factor analysis), (d) Parceling, (e) Structural regression models (including exploratory SEM), (f) Multi-group SEM (including measurement invariance testing), and (g) Indirect effects and moderation models.

The workshop comprises lectures and practical sessions, in which participants put into practice basic and more advanced SEM with PIAAC data. Participants primarily used the statistical software Mplus; yet, code and syntax for AMOS and R (lavaan) were provided.

Analyzing PIAAC data with multi-level analysis in Stata
Dozenten: Dr. Jan Paul Heisig (WZB, Berlin); FDZ PIAAC (GESIS Mannheim) Datum: 03. bis 04. April 2017 Ort: GESIS Mannheim Abstract: The first part of the workshop focuses on the analysis of PIAAC using the statistics package Stata. Emphasis was on two features of the PIAAC data that lead to challenges for the analyst: 1) the availability of multiple (10) “plausible values” for individual competence scores and 2) the use of jackknife replication methods for variance estimation. Different approaches to accounting for these features are presented. Participants were introduced to the piaactools package developed by the Polish PIAAC team, a convenient option that is, however, compatible only with a limited number of (regression) methods. Participants also have learned more flexible strategies for correctly estimating quantities that are not supported by piaactools (e.g., average marginal/partial effects). The second part of the workshop reviewed different approaches to analyzing multilevel data (mixed models, clustered standard errors, two-step procedures), with the emphasis again being on PIAAC and thus on country comparisons. Advantages and disadvantages of the different approaches and their implementation in Stata were discussed.

Analyzing PIAAC data with multi-level analysis in Stata

Dozenten: Dr. Jan Paul Heisig (WZB, Berlin); FDZ PIAAC (GESIS Mannheim)

Datum: 03. bis 04. April 2017

Ort: GESIS Mannheim

Abstract: The first part of the workshop focuses on the analysis of PIAAC using the statistics package Stata. Emphasis was on two features of the PIAAC data that lead to challenges for the analyst: 1) the availability of multiple (10) “plausible values” for individual competence scores and 2) the use of jackknife replication methods for variance estimation. Different approaches to accounting for these features are presented. Participants were introduced to the piaactools package developed by the Polish PIAAC team, a convenient option that is, however, compatible only with a limited number of (regression) methods. Participants also have learned more flexible strategies for correctly estimating quantities that are not supported by piaactools (e.g., average marginal/partial effects). The second part of the workshop reviewed different approaches to analyzing multilevel data (mixed models, clustered standard errors, two-step procedures), with the emphasis again being on PIAAC and thus on country comparisons. Advantages and disadvantages of the different approaches and their implementation in Stata were discussed.

Analyzing PIAAC log file data
Dozenten: Prof. Dr. Frank Goldhammer (DIPF); Krisztina Tóth (DIPF); FDZ PIAAC (GESIS Mannheim) Datum: 07. April 2017 Ort: GESIS Mannheim Abstract: The PIAAC 2012 study was the first fully computer-based large scale assessment in education. Using computers allowed not only to deliver innovative item formats and an adaptive test design, but also to collect a stream of user events (e.g., mouse clicks, text input) stored by the assessment system in log files. This data is interesting from a measurement point of view (e.g., to assess the quality of the response data), but also to address substantive research questions (e.g., to investigate the cognitive solution process). The process data gathered in PIAAC 2012 was made available for researchers by the OECD in 2017. Therefore, this workshop made participants familiar with the accessibility, structure and content of PIAAC log file data. In particular, the PIAAC LogDataAnalyzer was presented, that allows to extract log data from PIAAC xml log files. Users can select among pre-defined generic and task-specific aggregate variables (e.g., the number and sequence of page visits) and export them into a wide format. Furthermore, complete log data can be transformed and exported into a long format. The workshop also included sample analysis to demonstrate how exported log data can be further processed in standard statistical software such as the R environment or Weka.

Analyzing PIAAC log file data

Dozenten: Prof. Dr. Frank Goldhammer (DIPF); Krisztina Tóth (DIPF); FDZ PIAAC (GESIS Mannheim)

Datum: 07. April 2017

Ort: GESIS Mannheim

Abstract: The PIAAC 2012 study was the first fully computer-based large scale assessment in education. Using computers allowed not only to deliver innovative item formats and an adaptive test design, but also to collect a stream of user events (e.g., mouse clicks, text input) stored by the assessment system in log files. This data is interesting from a measurement point of view (e.g., to assess the quality of the response data), but also to address substantive research questions (e.g., to investigate the cognitive solution process). The process data gathered in PIAAC 2012 was made available for researchers by the OECD in 2017. Therefore, this workshop made participants familiar with the accessibility, structure and content of PIAAC log file data. In particular, the PIAAC LogDataAnalyzer was presented, that allows to extract log data from PIAAC xml log files. Users can select among pre-defined generic and task-specific aggregate variables (e.g., the number and sequence of page visits) and export them into a wide format. Furthermore, complete log data can be transformed and exported into a long format. The workshop also included sample analysis to demonstrate how exported log data can be further processed in standard statistical software such as the R environment or Weka.