CodeSCAN

ScreenCast ANalysis for Video Programming Tutorials

Alexander Naumann, Felix Hertlein, Jacqueline Höllig, Lucas Cazzonelli and Steffen Thoma
FZI Research Center for Information Technology, Karlsruhe, Germany
Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
Paper Code Data

Abstract

Programming tutorials in the form of coding screencasts play a crucial role in programming education, serving both novices and experienced developers. However, the video format of these tutorials presents a challenge due to the difficulty of searching for and within videos.

Addressing the absence of large-scale and diverse datasets for screencast analysis, we introduce the CodeSCAN dataset. It comprises 12,000 screenshots captured from the Visual Studio Code environment during development, featuring 24 programming languages, 25 fonts, and over 90 distinct themes, in addition to diverse layout changes and realistic user interactions.

Moreover, we conduct detailed quantitative and qualitative evaluations to benchmark the performance of Integrated Development Environment (IDE) element detection, color-to-black-and-white conversion, and Optical Character Recognition (OCR). We hope that our contributions facilitate more research in coding screencast analysis, and we make the source code for creating the dataset and the benchmark publicly available on this website.

Dataset Details

Currently, there is no big and fully annotated datasets publicly available. Bao et al. [1] provide a dataset, however, the annotations are currently not publicly available. Bergh et al. [2] published the, to the best of our knowledge, only dataset that is available. They focus on Python and Java and only classify images according to their content (i.e. no code, handwritten code, visible typset code and partially visible typeset code).

Our dataset was scraped from github.dev using Selenium.
  • 24 popular programming languages (according to Github)
  • 100 random repositories per language (with MIT, BSD-3 or WTFPL License), i.e. 2.400 repositories in total
  • Per repository we use 5 files, i.e. 12.000 files in total
  • 100 different themes and 25 different fonts
  • Diverse layouts changes, such as menu bar visibility, sidebar position, output window content, etc.
  • Numerous realistic interactions such as searching, typing and selecting within a file, etc.

Annotations

We can retrieve any type of visual annotations automatically.
Currently, we annotate: sidebar, active tab, output panel, code line number bar, coding grid (character width and line height), etc.

Binary Image

We have a high quality binary image for each screenshot, which is an ideal input for OCR engines.

Gallery

Randomly sampled impressions from the dataset.


References

[1] Bao, Lingfeng, Zhenchang Xing, Xin Xia, David Lo, Minghui Wu, and Xiaohu Yang. “Psc2code: Denoising Code Extraction from Programming Screencasts.” ACM Transactions on Software Engineering and Methodology 29, no. 3 (July 31, 2020): 1–38. https://doi.org/10.1145/3392093.

[2] Bergh, Adrienne, Paul Harnack, Abigail Atchison, Jordan Ott, Elia Eiroa-Lledo, and Erik Linstead. “A Curated Set of Labeled Code Tutorial Images for Deep Learning.” In 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA), 1276–80, 2020. https://doi.org/10.1109/ICMLA51294.2020.00200.