计算机与现代化 ›› 2021, Vol. 0 ›› Issue (08): 46-51.

• 图像处理 • 上一篇    下一篇

基于视觉特征的非侵入式用户界面输入项识别

  

  1. (南京航空航天大学计算机科学与技术学院, 江苏南京211106)
  • 出版日期:2021-08-19 发布日期:2021-08-19
  • 作者简介:王岩(1992—),男,江苏宿迁人,硕士研究生,研究方向:软件分析与测试,E-mail: 841331749@qq.com; 钱巨(1981—),男,江苏扬州人,副教授,硕士生导师,博士,研究方向:软件分析与测试。
  • 基金资助:
    “十三五”装备预研共用技术项目(41402020501-170441402030) 

Non-intrusive GUI Input Item Recognition Based on Visual Feature

  1. (College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China)
  • Online:2021-08-19 Published:2021-08-19

摘要: 识别应用界面中的输入项是界面测试数据自动生成的基础。现有界面输入项信息获取依赖被测对象所在系统提供的信息,无法用于非侵入式测试场合。本文从应用界面的视觉图像出发,提出一种基于视觉特征的非侵入式用户界面输入项识别方法。该方法从外部摄像头等途径获得的应用界面图片出发,利用直线检测和轮廓检测算法得到候选输入项列表,基于视觉特征,利用支持向量机(SVM)模型来对候选输入项进行判别,能够在不依赖被测目标底层系统的情况下获取输入项信息。实验结果表明,本文方法能够基本准确地识别界面图片中的输入项,具有一定的有效性。

关键词: 测试数据, 输入项识别, 视觉特征, 非侵入式测试

Abstract: Recognizing input items in an application interface is a basis of automatic GUI test data generation. How to get information of input items in existing interface depends on the information provided by the system of the object under test. They cannot be used under non-intrusive scenarios. This paper proposes a non-intrusive GUI input item recognition method based on visual features. The method detects input items in a GUI interface from GUI images obtained from sources like external cameras. It first uses line detection and contour detection algorithms to obtain a list of candidate input items, then uses support vector machine (SVM) to determine which of these candidates are real input items based on visual features. The method does not rely on the underlying system of the target under test. Experimental results show that it can recognize the input items in a GUI interface with a good accuracy and hence is of effectiveness.

Key words: test data, input item recognition, visual feature, non-intrusive testing