Chulwoo Pack

2025

1. AAAI

   Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract)

   Harsh Dubey, and Chulwoo Pack

   In Proceedings of the AAAI Conference on Artificial Intelligence, 2025

   Abs HTML

   To address the limitations of current Large-scale Video-Language Models (LVLMs) in fine-grained understanding and long-term temporal memory, we propose a novel video understanding approach that integrates a Vision Language Model (VLM) and a Large Language Model (LLM) with a textual memory mechanism to ensure continuity and contextual coherence. In addition, we introduce a novel evaluation metric, VAD-Score (Video Automated Description Score), to assess precision, recall, and F1 scores for events, subjects, and objects. Our approach delivers competitive results on a diverse set of videos from the DREAM-1K dataset, spanning categories such as live-action, animation, shorts, stock, and YouTube, with a focus on fine-grained comprehension.

2. ACR

   PEARL: Perceptual and Analytical Representation Learning for Video Anomaly Detection

   Omeshamisu Anigala, Kwanghee Won, and Chulwoo Pack

   SIGAPP Appl. Comput. Rev., Apr 2025

   Abs HTML

   Video anomaly detection is crucial for applications like surveillance and autonomous systems. Traditional methods often rely solely on visual cues, missing valuable contextual data. This paper presents Perceptual and Analytical Representation Learning (PEARL), a novel method that combines perceptual (raw sensory input) and analytical (higher-level context) modalities. Specifically, we integrate visual information with object tracking data, along with the tracking data-specialized normalization method, DOT-Norm, leveraging ID switching to capture high-level contexts of abnormal movements. We evaluate early- and late-fusion strategies to enhance anomaly detection, particularly for irregular movements marked by frequent track ID switches. Our approach, tested on the UCSD-Ped1 dataset, outperforms the state-of-the-art by improving precision (+0.082), recall (+0.104), F1 score (+0.149), and AUC (+0.053). These findings highlight the potential of integrating analytical tracking data with perceptual video frames in a multimodal learning approach for anomaly detection, paving the way for future applications and research where knowledge-driven analytical modalities are crucial.

2024

1. IJDAR

   Perceptual cue-guided adaptive image downscaling for enhanced semantic segmentation on large document images

   Chulwoo Pack, Leen-Kiat Soh, and Elizabeth Lorang

   International Journal on Document Analysis and Recognition (IJDAR), Apr 2024

   Abs HTML Code

   Image downscaling is an essential operation to reduce spatial complexity for various applications and is becoming increasingly important due to the growing number of solutions that rely on memory-intensive approaches, such as applying deep convolutional neural networks to semantic segmentation tasks on large images. Although conventional content-independent image downscaling can efficiently reduce complexity, it is vulnerable to losing perceptual details, which are important to preserve. Alternatively, existing content-aware downscaling severely distorts spatial structure and is not effectively applicable for segmentation tasks involving document images. In this paper, we propose a novel image downscaling approach that combines the strengths of both content-independent and content-aware strategies. The approach limits the sampling space per the content-independent strategy, adaptively relocating such sampled pixel points, and amplifying their intensities based on the local gradient and texture via the content-aware strategy. To demonstrate its effectiveness, we plug our adaptive downscaling method into a deep learning-based document image segmentation pipeline and evaluate the performance improvement. We perform the evaluation on three publicly available historical newspaper digital collections with differences in quality and quantity, comparing our method with one widely used downscaling method, Lanczos. We further demonstrate the robustness of the proposed method by using three different training scenarios: stand-alone, image-pyramid, and augmentation. The results show that training a deep convolutional neural network using images generated by the proposed method outperforms Lanczos, which relies on only content-independent strategies.

2022

1. JOCCH

   Augmentation-based Pseudo-Ground truth Generation for Deep Learning in Historical Document Segmentation for Greater Levels of Archival Description and Access

   Chulwoo Pack, Yi Liu, Leen-Kiat Soh, and 1 more author

   Journal on Computing and Cultural Heritage (JOCCH), Apr 2022

   Abs HTML

   The successful use of deep learning solutions for document image segmentation typically relies on a large number of manually labeled ground truth examples, which is expensive to obtain for historical document images that have significant noise effects and variation. At the same time, successful applications of deep learning solutions for document image segmentation have rich potential to facilitate greater levels of description in archival collections (e.g., at and below the item-level). These greater levels of description are critical to increasing access and use of archival collections across an array of research domains. In response, this article investigates whether an augmentation-based approach to generating pseudo-ground truth can be effective with a limited number of labeled images in a document segmentation application. The rationale is that if we can decrease the cost of generating ground truth through augmentation-based approaches, we can use these approaches as part of the description and access pipelines for historical library and archival collections. In this initial exploration, we first generate synthetic images and corresponding pseudo-ground truth using a set of existing degradation-based augmentation models from a small number of labeled actual images. When generating synthetic images, we control the visual quality distortion based on OCR word-level confidence to avoid generating images unlikely to be present in the dataset. Then, we perform several investigations to examine the impact of incorporating pseudo-ground truth data in the training of the deep learning network dhSegment and further evaluate the use of multiple combinations of degradation models. We also assess the generalizability of the approach by applying the trained network on a larger dataset. Our investigations primarily use real-world datasets known to have significant noise effects. Results show that augmentation-based pseudo-ground truth generation is capable of improving segmentation performance with the use of the full original dataset and requires only 30% of the original dataset. Results also show that using more than three degradation models is likely to cause overfitting during training. Furthermore, we show that a segmentation network trained on pseudo-ground truth data has generalization capability.

2021

1. JEI

   Visual domain knowledge-based multimodal zoning for textual region localization in noisy historical document images

   Chulwoo Pack, Leen-Kiat Soh, and Elizabeth Lorang

   Journal of Electronic Imaging, Apr 2021

   Abs HTML Code

   Document layout analysis, or zoning, is important for textual content analysis such as optical character recognition. Zoning document images such as digitized historical newspaper pages are challenging due to noise and quality of the document images. Recently, effective data-driven approaches, such as leveraging deep learning, have been proposed, albeit with the concern of requiring larger training data and thus incurring additional cost of ground truthing. We propose a zoning solution by incorporating a knowledge-driven document representation, gravity map, into a multimodal deep learning framework to reduce the amount of time and data required for training. We first generate a gravity map for each image, considering the centroid distance and area between a cell in a Voronoi tessellation and its content to encode visual domain knowledge of a zoning task. Second, we inject the gravity maps into a deep convolution neural network (DCNN) during training, as an additional modality to boost performance. We report on two investigations using two state-of-the-art DCNN architectures and three datasets: two sets of historical newspapers and a set of born-digital contemporary documents. Evaluations show that our solution achieved comparable segmentation accuracy using fewer training epochs and less training data compared to a naïve training scheme.