EXPERIENCE

Principal Data Scientist, Phenom People, 2019-present

Using AI and Data Science to help 1B people find the right job.

Director, Data Science, 2U, Inc., 2015-2019

Developed data-driven algorithms for digital learning: to describe fundamental limitations on discovery of domain models; to align noisy log files to expected event schedules; to define a computational model of course content and learning outcomes. Primarily working in R, these inventions led to internal products and scholarly publications.

Provided organizational guidance on data science, learning science and educational technology. Worked with product management to design new feature agenda, and to articulate market positioning and differentiation.

Research Scientist, Center for Digital Data, Analytics & Adaptive Learning, Pearson, Inc., 2013-2015

Developed data-driven algorithms for digital learning: ways to probabilistically estimate student mastery of a skill, to discover prerequisite skill relationships, to flag learning activities for editorial attention, and others. Primarily working in R, these inventions led to internal prototypes, patent applications and scholarly publications.

Planned and supervised field-testing of Pearson technologies with hundreds of K-12 students in Utah and Michigan, and derived insights from data that drove product revisions.

Leveraged academic background and relationships to bring outside expertise to Pearson projects, to support colleagues'​ professional development, and to enhance Pearson's position as a thought leaders in educational technology.

Post-doctoral Fellow, Carnegie Mellon University, 2011-2013

Studied the effect of hints on learning by creating novel statistical models for data from tutoring software, and by designing and conducting in-school studies. Presented research results at academic conferences.

Data Administrator, Magee-Womens Research Institute and Foundation, 2010-2011

Processed hundreds of gigabytes of data for biomedical informatics tasks using custom and third-party software and scripting in a Unix environment.

Research Assistant, JustSystems Evans Research, Inc., 2006-2009

Designed, developed and evaluated software and interfaces in machine learning, recommender systems, semantic modeling, visualization, natural language processing and ontologies.

Research & Development Intern, McCabe & Associates, 1997-1998

Worked on software quality assurance, including unit, regression and function testing.

FULL-LENGTH REFEREED PUBLICATIONS

Goldin, I., Galyardt, A. (2018) Most of the time, it works every time: Limitations in using learning curve analysis to refine domain models. Journal of Educational Data Mining, Vol. 10(2), pp. 55-92

Galyardt, A., Goldin, I. (2015). Move your lamp post: Recent data reflects learner knowledge better than older data. Journal of Educational Data Mining, Vol. 7(2), pp. 83-108

Galyardt, A., Goldin, I. (2015). Evaluating simplicial mixtures of Markov chains for modeling student metacognitive strategies. Quantitative Psychology Research, Vol. 89, pp. 377–393

Goldin, I., Pinkus, R.L., Ashley, K. (2015) Validity and reliability of an instrument for assessing case analyses in Bioengineering Ethics education. Science and Engineering Ethics, Vol. 21(3), pp. 789–807

Goldin, I., Renken, M., Galyardt, A., Litkowski, E. (2014) Individual differences in identifying sources of science knowledge. Proceedings of 9th European Conference On Technology Enhanced Learning

Almond, R. G., Goldin, I., Guo, Y., Wang, N. (2014). Vertical and stationary scales for progress maps. Proceedings of 7th International Conference on Educational Data Mining

Goldin, I., Carlson, R. (2013) Learner differences and hint content. 16th International Conference on Artificial Intelligence and Education

Goldin, I., Ashley, K. (2012) Eliciting formative assessment in peer review. Journal of Writing Research, 4(2), pp. 203-237.

Goldin, I., Koedinger, K., Aleven, V. (2012) Learner differences in hint processing. Proceedings of the 5th International Conference on Educational Data Mining. [Finalist for Best Paper Award]

Ashley, K., Goldin, I. (2011) Toward AI-enhanced computer-supported peer review in legal education. Proceedings of the 24th International Conference on Legal Knowledge and Information Systems

Goldin, I., Ashley, K. (2011) Peering inside peer review with Bayesian models. Proceedings of the 15th International Conference on Artificial Intelligence and Education

Goldin, I., Ashley, K. (2010) Eliciting informative feedback in peer review: importance of problem-specific scaffolding. Proceedings of the 10th International Conference on Intelligent Tutoring Systems

Goldin, I., Ashley, K., Pinkus, R.L. (2006) Assessing case analyses in Bioengineering Ethics education: Reliability and training. Proceedings of the 8th International Conference on Engineering Education

Goldin, I., Ashley, K., Pinkus, R.L. (2001) Introducing PETE: computer support for teaching ethics. Proceedings of the 8th International Conference on Artificial Intelligence & Law

ADDITIONAL PUBLICATIONS

Goldin, I., Narciss, S., Foltz, P., Bauer, M. (2017). New Directions in Formative Feedback in Interactive Learning Environments. International Journal of Artificial Intelligence in Education.

Goldin, I., Pavlik Jr, P., Ritter, S. (2016). Discovering domain models in learning curve data. Design Recommendations for Intelligent Tutoring Systems: Domain Modeling (Vol 4). US Army Research Laboratory

Goldin, I., Galyardt, A. (2015). Convergent validity of a student model: Recent-Performance Factors Analysis. Proceedings of 8th International Conference on Educational Data Mining (EDM)

Goldin, I., Galyardt, A. (2015). On Feasibility of Using Learning Curve Analysis to Refine Domain Models. Meeting of the Advisory Board of the Generalized Intelligent Framework for Tutoring

Goldin, I., Galyardt, A. (2015). Viz-R: Using Recency to Improve Student and Domain Models. Proceedings of 2nd ACM Conference on Learning at Scale

Forsgren Velasquez, N., Goldin, I., Martin, T., Maughan, J. (2014). Learning aid use patterns and their impact on exam performance in online developmental mathematics. Proceedings of 7th International Conference on Educational Data Mining (EDM)

Galyardt, A., Goldin, I. (2014). Recent-Performance Factors Analysis. Proceedings of 7th International Conference on Educational Data Mining (EDM)

Scheines, R., Silver, E., Goldin, I. (2014). Discovering prerequisite relationships among knowledge components. Proceedings of 7th International Conference on Educational Data Mining (EDM)

Goldin, I., McLaren, B. (2014). Towards instrumenting collaborative learning and assessment in the digital ocean. Proceedings of the Workshop on Social, Motivational and Affective Dimensions of Learning through Social Interaction, at the 11th International Conference of the Learning Sciences (ICLS)

Goldin, I., McLaren, B. (2014). Towards instrumenting collaborative learning and assessment in the digital ocean. Proceedings of 3rd Workshop on Intelligent Support for Learning in Groups, at the 12th International Conference on Intelligent Tutoring Systems (ITS)

Renken, M., Litkowski, E., Goldin, I., Oliver, M. (2014). Students’ Recognition of the Value of Scientific Process When Evaluating Science Explanations. Presented at the American Psychological Association Annual Conference (APA)

Goldin, I. (2013) Towards Supporting Students and Instructors with Models of Peer Assessment. Invited symposium. 78th Annual Meeting of the Psychometric Society (IMPS)

Galyardt, A., Goldin, I. (2013). Modeling Student Metacognitive Strategies in an Intelligent Tutoring System. Presented at 78th Annual Meeting of the Psychometric Society (IMPS)

Goldin, I., Koedinger, K., Aleven, V. (2013) Hints: You Can't Have Just One. 6th International Conference on Educational Data Mining (EDM)

Goldin, I. (2013) Examining Peer Reviewer Calibration with Bayesian Models. It's About Time: Addressing the Many Challenges of Analyzing Multi-Scale Temporal Data. Workshop at Alpine Rendez-Vous

Goldin, I., Carlson, R. (2013) Learner Differences and Hint Content. Data Analysis and Interpretation for Learning Environments (DAILE). Workshop at Alpine Rendez-Vous

Goldin, I., Ashley, K., Schunn, C. (2012) Redesigning Educational Peer Review Interactions Using Computer Tools: An Introduction. Journal of Writing Research, 4(2), 111-119.

Goldin, I. (2012) Accounting for Peer Reviewer Bias with Bayesian Models. Proceedings of the Workshop on Intelligent Support for Learning Groups at the 11th International Conference on Intelligent Tutoring Systems

Ashley, K., Goldin, I. (2012) Computer-Supported Peer Review in a Law School Context. U. of Pittsburgh Legal Studies Research Paper No. 2012-24.

Goldin, I (2011) A focus on content: The use of rubrics in peer review to guide students and instructors. PhD dissertation, University of Pittsburgh.

Goldin, I., Ashley, K., Pinkus, R.L. (2006) Teaching case analysis through framing: Prospects for an ITS in an ill-defined domain. Proceedings of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains at the 8th International Conference on Intelligent Tutoring Systems

Goldin, I., Chapman, W.W. Learning to detect negation with ‘not’ in medical texts. Proceedings of the Workshop on Text Analysis and Search for Bioinformatics at the 26th International ACM SIGIR Conference

PATENT APPLICATIONS

Mahe Bayireddi, Hari Bayireddy, Sivanand Akella, Suresh Babu Devineni, S.S.S. Venkateswara Rao Alapati, Shivaramakrishna Nyshadham, Ilya Goldin. "Knowledge engine using machine learning and predictive modeling for optimizing recruitment management systems". U.S. Patent Application US16/843,805 (International PCT US20-27329).

Gonzalez-Brenes, J., Goldin, I.M., Larusson, J.A., Behrens, J., McTavish, T., Rho, Y.J., Anderson, J.M., Kukartsev, G.A. "Content Database Generation". U.S. Patent Grant US 10,713,225 B2

Gonzalez-Brenes, J., Goldin, I.M., Larusson, J.A., Behrens, J., McTavish, T. "System and method for increasing data transmission rates through a content distribution network with customized aggregations". U.S. Patent Grant US 10,027,740 B2

Scheines, R.; Silver, E.; Goldin, I. "Discovering Prerequisite Relationships among Knowledge Components." U.S. Patent Application 62/188,184

Evans, D.A.; Bennett, J.; Hull, D.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Anchoring Expressions Based on an Ontological Model of Semantic Information.” U.S. Patent Application 20080294426

Hull, D.; Evans, D.A.; Bennett, J.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing Semantic Update and Replace Operations.” U.S. Patent Application 20080294425

Evans, D.A.; Bennett, J.; Hull, D.; Cheng, H.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing Semantically Informed Text Operations.” U.S. Patent Application 20080295013

Cheng, H.; Evans, D.A.; Bennett, J.; Hull, D.; Qu, Y.; Tenny, C.; Montgomery, J.; Goldin, I. “Method and Apparatus for Performing a Semantically Informed Merge Operation.” U.S. Patent Application 20080294427

Qu, Y.; Evans, D.A.; Goldin, I. “Method and Apparatus for the Automated Construction of Models of Activities from Textual Descriptions of the Activities.” U.S. Patent Application 20080294398

RESEARCH GRANTS

Community Care Behavioral Health Organization. Analysis of Disengagement in Mental Health Provider Networks. PI, 2011, $12,000.

Provost's Advisory Council on Instructional Excellence, University of Pittsburgh. A Peer Review-Based Student Model for Ill-Defined Problem-Solving. Co-author with K. Ashley, 2009, $25,000.

PUBLIC SOFTWARE

Curtis, S.M. (author), Goldin, I. (contributor). (2012) mcmcplots: visualization of Markov-Chain Monte Carlo output for Bayesian modeling. http://cran.r-project.org/web/packages/mcmcplots

Goldin, I. (2012) ProfHelp and ProfHelp-ID: multilevel Bayesian models of learner differences with hint processing.

Goldin, I. (2012) Peer reviewer bias: multilevel Bayesian models of bias in peer assessment and bias-instructor score relationship.

Goldin, I. (2011) Rubrics in peer review: multilevel Bayesian models of multidimensional rubrics in peer review.

Goldin, I. (2010) Comrade: web-based peer review software. Used in 4 studies by 200 students and 3 instructors.

Goldin, I. (2001) Professional Ethics Tutoring Environment (PETE): web-based ethics tutoring software.

INVITED TALKS

Data-Driven Curriculum Design for Medical Education. National Academy of Medicine SIG on Education of the Health Care and Science Workforce (2018)

Applications of Data Science in Education. Ontario Institute for Studies in Education, University of Toronto (2016)

Applications of Data Science in Education. Centre for Teaching, Learning and Technology, University of British Columbia (2016)

Domain Modeling Using Learning Curve Analysis. Learning Analytics Summer Institute-local, New York University (2016)

On Formative Assessment in Interactive Learning Environments. Learning Sciences Institute, Arizona State University (2013)

Peering into Peer Review with Bayesian Models. Institute for Language, Cognition and Computation, University of Edinburgh, UK (2013)

Peering into Peer Review with Bayesian Models. Institute of Educational Technology, The Open University, UK (2013)

Towards Effective Feedback for All: Learner Differences in Hint Processing. Learning Analytics Seminar Series, Teachers College, Columbia University (2013)

Towards Effective Feedback for All: Learner Differences in Hint Processing. Artificial Intelligence Forum, Intelligent Systems Program, University of Pittsburgh (2012)

ACADEMIC SERVICE

Guest Co-editor

International Journal of Artificial Intelligence in Education, Special Issue: Formative Feedback in Interactive Learning Environments (2017), with Foltz, P., Narciss, S., Bauer, M.

Journal of Writing Research, Special issue: Redesigning Peer Review Interactions Using Computer Tools (2012), with Ashley, K., Schunn, C.

Program Committee Member

International Conference on Intelligent User Interfaces (2016)

International Conference on Artificial Intelligence in Education (2015-2018)

International Conference on Educational Data Mining (2013-2019)

Artificial Intelligence and Feedback, a workshop at International Joint Conferences on Artificial Intelligence (2015)

Data Mining for Educational Assessment and Feedback (ASSESS), a workshop at KDD 2014

International Conference on Creating, Connecting and Collaborating through Computing (2011)

Journal Reviewer

International Journal of Artificial Intelligence in Education (2016, 2015, 2013)

IEEE Transactions on Learning Technologies (2018, 2013)

Journal of Educational Data Mining (2018, 2011)

Technology, Instruction, Cognition and Learning (2013)

Journal of Educational Psychology (2012)

Conference Reviewer

ACM CHI Conference on Human Factors in Computing Systems (2014)

Society for Research on Educational Effectiveness (Spring 2013)

International Conference on Artificial Intelligence in Education (2011)

International Conference on Educational Data Mining (2012-14, 2010)

International Conference on Computers and Education (2009)

Conference of Florida Artificial Intelligence Research Society (2009, 2008)

Grant Application Reviewer

National Science Foundation, USA (2013)

Economic and Social Research Council, UK (2008)

Chair

Workshops, International Conference on Educational Data Mining (2016)

Interactive Events track, International Conference on Artificial Intelligence in Education (2015), with Olga Santos

Intelligent Support for Learning in Groups, a workshop at International Conference on Intelligent Tutoring Systems (2014) and International Conference on Artificial Intelligence in Education (2015), with Roberto Maltinez-Maldonado, Erin Walker, Jihie Kim, and Cindy Hmelo-Silver

Formative Feedback in Interactive Learning Environments, a workshop at International Conference on Artificial Intelligence and Education (2013), with Taylor Martin, Ryan Baker, Vincent Aleven and Tiffany Barnes

Crosspollination: Multidisciplinary Approaches to Human Cognition and Behavior, a workshop at inter-Science of Learning Centers (iSLC) Conference (2012), with Amy Ogan and So-One Hwang

Computer-Supported Peer Review in Education: Synergies with Intelligent Tutoring Systems, a workshop at International Conference on Intelligent Tutoring Systems (2010)

Committee Member

Doctoral dissertation committee, University of Southern California, EdD candidate Erika Maldonado (2018)

Internal Volunteering

Liaison to Executive Committee of the Pittsburgh Science of Learning Center (PSLC) from PSLC postdoctoral fellows (2012-13)

MENTORING

Laura Malkiewich, intern, 2U, Inc.

Yetian Chen, intern, Pearson Education, with Jose Gonzalez-Brenes

Ryan Carlson, Ph.D. student, Carnegie Mellon University, with Kenneth Koedinger and Carolyn Rose

LearnLab Summer School: Educational Data Mining track, Pittsburgh Science of Learning Center (2012, 2013)

Timothy Hirsh, intern, JustSystems Evans Research, Inc.