HCI Overview Week 41 - Evaluating User Interfaces 2 Empirical & JavaScript for Desktop Apps

TRACK - A

---

1. What’s an Experiment in HCI and When is It Used?

Definition: An experiment in HCI is a scientifically rigorous method used to measure the performance and impact of different variables on user interactions with interfaces. It involves manipulating one or more independent variables to observe their effect on dependent variables.

When to Use: Experiments are employed when you need to:

• Test specific hypotheses about user behavior or interface effectiveness.
• Measure the impact of design changes (e.g., a new search feature) on user performance metrics like task completion time or accuracy.
• Validate whether a new interface improves usability compared to an existing one.

Example from Lecture Notes: Evaluating a new email search interface by comparing search completion times and accuracy between the new and old interfaces.

---

2. Key Concerns: Validity and Reliability. What Are Those and Why Those?

Validity:

• Definition: Refers to the accuracy and credibility of the experiment, ensuring that the study measures what it intends to measure.
• Importance: Ensures that the conclusions drawn are based on sound evidence and that the results truly reflect the relationship between variables.
• Types:
  • Internal Validity: The degree to which the experiment accurately establishes a cause-effect relationship between variables, free from confounding factors.
  • External Validity: The extent to which the findings can be generalized to other settings, populations, or times.

Reliability:

• Definition: Pertains to the consistency of the experiment results over repeated trials.
• Importance: Ensures that the findings are dependable and can be replicated under similar conditions.
• Enhancing Reliability:
  • Using standardized procedures.
  • Ensuring consistent measurement techniques.
  • Controlling environmental variables.

Why They Matter:

• Validity ensures the experiment’s findings are meaningful and accurate.
• Reliability ensures that results are consistent and trustworthy, allowing for replication and verification.

---

3. What Are the Constituents of an Experiment?

An experiment in HCI typically comprises the following components:

1. Research Questions and Hypotheses:
   • Define what you aim to investigate and predict the relationship between variables.
2. Independent Variables (IVs):
   • Factors manipulated by the experimenter (e.g., type of search interface).
3. Dependent Variables (DVs):
   • Outcomes measured to assess the effect of IVs (e.g., task completion time, accuracy).
4. Participants:
   • Individuals who take part in the experiment, representing the target user population.
5. Experimental Design:
   • Structure outlining how IVs and DVs are organized (e.g., within-participants or between-participants design).
6. Control of Nuisance Factors:
   • Methods to minimize the impact of external variables that could influence the results (e.g., random assignment, counterbalancing).
7. Data Collection Methods:
   • Techniques for gathering quantitative and qualitative data (e.g., task performance metrics, verbal protocols).
8. Data Analysis:
   • Processes for interpreting the collected data using descriptive and inferential statistics.
9. Ethical Considerations:
   • Ensuring participant rights, informed consent, and confidentiality.

---

4. What’s the Role of the Hypothesis and How Is It Formulated?

Role of the Hypothesis:

• Purpose: Acts as a predicted relationship between independent and dependent variables, guiding the direction and focus of the experiment.
• Function: Provides a clear statement that can be tested and either supported or refuted through empirical data.

Formulation:

1. Identify Variables:
   • Determine which IVs and DVs are involved based on research questions.
2. Predict Relationship:
   • State how you expect the IVs to influence the DVs.
3. Structure:
   • Typically follows an “If [IV], then [DV]” format.

Example from Lecture Notes:

• Research Question: Does the new search interface improve search efficiency?
• Hypothesis: Users will complete searches faster and with higher accuracy using the new search interface compared to the old one.

Types of Hypotheses:

• Confirmatory: Tests specific, predefined predictions.
• Exploratory: Aims to discover new insights without specific predictions.

---

5. What Are Independent and Dependent Variables?

Independent Variables (IVs):

• Definition: Factors manipulated by the experimenter to observe their effect.
• Examples:
  • Type of user interface (new vs. old).
  • User expertise level.
  • Type of feedback provided.

Dependent Variables (DVs):

• Definition: Outcomes measured to assess the impact of IVs.
• Examples:
  • Task completion time.
  • Accuracy or error rates.
  • User satisfaction scores.

Key Points:

• IVs are causes or inputs.
• DVs are effects or outputs that respond to IV changes.

---

6. How Are Dependent Variables Selected?

Selection Criteria:

1. Alignment with Research Questions:
   • Ensure DVs directly address the research objectives and hypotheses.
2. Operationalization:
   • Define how abstract concepts will be measured concretely (e.g., “search efficiency” measured by completion time and accuracy).
3. Relevance and Significance:
   • Choose DVs that provide meaningful insights into user performance and experience.
4. Feasibility:
   • Ensure that the DVs can be reliably and validly measured within the experimental setup.

Best Practices:

• Utilize multiple measures to enhance reliability and validity.
• Incorporate both quantitative (e.g., time, accuracy) and qualitative (e.g., user feedback) data when appropriate.

Example from Lecture Notes:

• For evaluating a new email search interface:
  • Completion Time: Quantitative measure of how long users take to find specific emails.
  • Accuracy: Quantitative measure of the correctness of search results.

---

7. How Are Participants Chosen?

Selection Process:

1. Define Target Population:
   • Identify characteristics of users relevant to the study (e.g., novice vs. expert users).
2. Sampling Method:
   • Avoid Convenience Sampling: Instead of selecting easily accessible participants (e.g., friends, colleagues), strive for a sample that represents the target population.
   • Representative Sampling: Ensure participants reflect the diversity and characteristics of actual users.
3. Sample Size:
   • Typical Size: HCI studies often use 12-20 participants.
   • Power Analysis: Conduct statistical methods to determine the appropriate sample size needed to detect meaningful effects (e.g., 64 participants per condition for medium-sized effects in a between-subjects design).
4. Recruitment:
   • Use strategies that reach the intended user demographic, such as online recruitment, user panels, or targeted advertisements.

Ethical Considerations:

• Informed Consent: Ensure participants understand the study’s purpose and procedures.
• Confidentiality: Protect participants’ personal information.
• Compensation: Provide fair compensation without being coercive.

Example from Lecture Notes:

• Conducting think-aloud studies with 12 participants to evaluate the usability of a gesture-controlled device.

---

8. How Are Results from Experiments Analysed?

Data Analysis in HCI Experiments:

1. Descriptive Statistics:
   • Purpose: Summarize and describe the main features of the data.
   • Methods:
     • Summary Statistics: Mean, median, variance.
     • Visualization: Histograms, scatter plots, line plots.
2. Inferential Statistics:
   • Purpose: Draw conclusions and test hypotheses about the population based on sample data.
   • Methods:
     • Hypothesis Testing: Determine if observed effects are statistically significant (e.g., p-values, t-tests).
     • Confidence Intervals: Estimate the range within which the true population parameter lies with a certain confidence level (e.g., 95%).
3. Explaining Results:
   • Link to Theories: Use existing frameworks to interpret findings.
   • Incorporate Qualitative Data: Integrate user feedback and observations to provide context.
   • Practical Implications: Discuss how results inform design improvements.

Best Practices:

• Use multiple measures to enhance the robustness of findings.
• Ensure statistical methods align with research questions and data types.

Example from Lecture Notes:

• Using inferential statistics to assess whether the new search interface significantly reduces task completion time compared to the old interface.

---

9. What Is the Difference Between Qualitative and Quantitative Data?

Qualitative Data:

• Nature: Descriptive and non-numerical.
• Purpose: Provides in-depth insights into user behaviors, thoughts, and experiences.
• Examples: Verbal protocols from think-aloud studies, user interviews, observations.
• Analysis: Thematic coding, content analysis, narrative descriptions.

Quantitative Data:

• Nature: Numerical and measurable.
• Purpose: Quantifies aspects of user interactions, enabling statistical analysis.
• Examples: Task completion times, error rates, survey ratings.
• Analysis: Statistical tests, correlation, regression analysis.

Key Differences:

• Data Type: Qualitative is textual or visual; quantitative is numerical.
• Analysis Methods: Qualitative relies on interpretative techniques; quantitative uses mathematical and statistical methods.
• Outcome: Qualitative offers depth and context; quantitative provides breadth and generalizability.

Example from Lecture Notes:

• Quantitative: Measuring the time taken to complete a search task.
• Qualitative: Analyzing participants’ verbalized thoughts during a think-aloud study to identify usability issues.

---

10. What Is a Field Study and How Is It Different from an Experiment?

Field Study:

Definition: A field study involves evaluating systems in their real-world settings, observing genuine user interactions within natural environments.

Types:

1. Field Evaluations of Prototypes:
   • Testing early versions of a system in real settings to gather preliminary feedback.
2. Pilot Studies:
   • Deploying a prototype or unfinished system to gather initial data for refinement.
3. Deployment Studies:
   • Releasing a near-final or final system to users and collecting long-term data.

Advantages:

• High Ecological Validity: Captures authentic user behaviors and interactions.
• Contextual Insights: Understands the social and organizational impacts of the system.
• Real Users and Tasks: Observes how the system functions with actual user motivations and activities.

Challenges:

• Loss of Control: Difficult to manage external variables and ensure consistency.
• Participant Recruitment: Finding and engaging representative users can be challenging.
• Data Collection: Requires unobtrusive methods to avoid influencing user behavior.

Differences from Experiments:

Aspect	Field Studies	Experiments
Setting	Real-world environments	Controlled, often artificial environments
Control Over Variables	Low; many external factors are present	High; variables are carefully controlled
Ecological Validity	High	Lower, due to artificial settings
Data Type	Often qualitative and observational	Primarily quantitative, but can include qualitative
Purpose	Understand system use in natural contexts	Test specific hypotheses under controlled conditions
Duration	Can be long-term	Typically shorter-term

Example from Lecture Notes: Deploying an AAC (Augmentative and Alternative Communication) application through an online marketplace and collecting usage data and user feedback over an extended period.

TRACK - B

---

1. How JavaScript Can Be Used to Develop Desktop Applications

JavaScript is traditionally associated with web development, running in browsers to create interactive web pages. However, with the advent of frameworks like Electron, NW.js, and Tauri, JavaScript can also be leveraged to build cross-platform desktop applications. Here’s how:

• Cross-Platform Compatibility: JavaScript-based desktop applications can run on Windows, macOS, and Linux without significant changes to the codebase.
• Web Technologies: By using HTML, CSS, and JavaScript, developers can create rich user interfaces. This approach leverages existing web development skills and tools.
• Access to Native APIs: Through integration with Node.js and other backend technologies, JavaScript applications can interact with the file system, hardware, and other native OS features.
• Packaging and Distribution: Tools like Electron package the application along with a browser engine (typically Chromium) and Node.js runtime, simplifying distribution and ensuring consistency across platforms.

Example Frameworks:

• Electron: Combines Chromium and Node.js to create desktop applications.
• NW.js: Similar to Electron, allowing the use of Node.js modules directly in the browser context.
• Tauri: A lightweight alternative focusing on security and smaller bundle sizes, using the system’s webview.

Popular Applications Built with JavaScript:

• Visual Studio Code: A powerful code editor by Microsoft.
• Slack: The desktop client for the popular communication tool.
• Discord: The widely-used chat application for gamers and communities.

2. Difference Between Running JavaScript in the Browser and with Electron

While both environments execute JavaScript, there are significant differences between running it in a browser and within an Electron application:

Environment and Context

• Browser:
  • Sandboxed Environment: JavaScript runs in a secure, restricted context, limiting access to the user’s system for security reasons.
  • Web APIs: Access to DOM, CSSOM, and browser-specific APIs for rendering and interacting with web pages.
  • Asynchronous Operations: Heavily relies on APIs like fetch, XMLHttpRequest, and WebSockets for network operations.
• Electron:
  • Node.js Integration: Electron combines Chromium’s rendering engine with Node.js, allowing JavaScript to access both browser APIs and Node.js APIs.
  • File System Access: Direct access to the file system, enabling reading, writing, and manipulating files without server-side components.
  • Inter-Process Communication (IPC): Facilitates communication between the main process (which controls the application lifecycle) and renderer processes (which handle the UI).

Capabilities and APIs

• Browser:
  • Limited to web standards and APIs defined by the browser (e.g., Web Storage, Service Workers).
  • Restricted access to system resources for security and privacy.
• Electron:
  • Full access to Node.js modules and native OS APIs, enabling functionalities like spawning child processes, accessing hardware, and integrating with native system features.
  • Ability to create native menus, dialogs, notifications, and other desktop-specific UI elements.

Performance and Resource Usage

• Browser:
  • Optimized for loading and rendering web content efficiently.
  • Generally lighter in terms of resource usage for web-specific tasks.
• Electron:
  • Bundles Chromium and Node.js, resulting in larger application sizes and higher memory consumption compared to native applications or lightweight web pages.
  • Potential for performance bottlenecks if not managed properly, especially with multiple renderer processes.

Security Considerations

• Browser:
  • Strict security models to prevent malicious scripts from compromising the system.
  • Regular updates and patches from browser vendors to address vulnerabilities.
• Electron:
  • Developers must implement their own security measures to prevent vulnerabilities, such as ensuring proper sandboxing, avoiding the execution of untrusted code, and managing permissions carefully.
  • The combination of Node.js and Chromium increases the attack surface, requiring diligent security practices.

3. Interplay Between Node.js and Electron in Creating Desktop Apps with JavaScript

Electron is essentially a framework that combines Chromium (for rendering the UI) and Node.js (for backend functionality) to enable the development of desktop applications using JavaScript, HTML, and CSS. Here’s how Node.js and Electron work together in this ecosystem:

Architecture Overview

• Main Process:
  • Runs Node.js and manages the lifecycle of the application.
  • Handles creating and managing browser windows (renderer processes), managing native interfaces, and performing system-level operations.
  • Can execute any Node.js code, including accessing the file system, spawning child processes, and interacting with native modules.
• Renderer Processes:
  • Each browser window runs in its own renderer process, which is essentially a Chromium instance.
  • Renderer processes handle the UI and user interactions, executing JavaScript that can interact with the DOM.
  • They have access to both browser APIs and a subset of Node.js APIs, allowing for rich interactions between the frontend and backend.

Communication Between Processes

• Inter-Process Communication (IPC):
  • Electron provides IPC modules (ipcMain and ipcRenderer) to facilitate communication between the main and renderer processes.
  • This allows the UI (renderer) to request actions from the backend (main), such as accessing the file system or performing network requests, while keeping sensitive operations isolated in the main process.

Integration of Node.js Features

• Node.js Modules:
  • Developers can utilize the vast ecosystem of Node.js modules within the main process to implement backend functionalities like database interactions, file handling, and network communications.
• Access to Native APIs:
  • Node.js enables access to native OS APIs, allowing Electron applications to perform tasks that are not possible with browser-based JavaScript alone.
• Synchronous and Asynchronous Operations:
  • Node.js provides both synchronous and asynchronous APIs, giving developers flexibility in handling operations that may block the main thread or run concurrently.

Development Workflow

1. Setup and Initialization:
   • An Electron application typically starts by running the main process, which initializes the application and creates the initial renderer window.
2. UI Rendering:
   • The renderer process loads the HTML/CSS/JS files to display the user interface, just like a web page.
3. Backend Operations:
   • When the UI needs to perform backend operations (e.g., reading a file), it sends an IPC message to the main process.
   • The main process handles the request using Node.js APIs and sends back the results via IPC.
4. State Management:
   • Electron applications often manage state between the main and renderer processes, ensuring synchronization and consistent behavior across the application.

Advantages of This Interplay

• Unified Development Stack:
  • Developers can use a single language (JavaScript) and a unified set of tools for both frontend and backend development.
• Reusability:
  • Code can be reused between the frontend (UI) and backend (application logic), improving efficiency and reducing duplication.
• Flexibility:
  • The combination allows for a wide range of functionalities, from simple utilities to complex, feature-rich applications.

Considerations and Best Practices

• Security:
  • Minimize the exposure of Node.js APIs in renderer processes to reduce the risk of exploitation.
  • Use context isolation and disable remote module access when possible.
• Performance:
  • Optimize renderer processes to prevent excessive memory usage and ensure smooth UI performance.
  • Manage the number of renderer processes carefully, as each consumes system resources.
• Modularity:
  • Structure the application into clear, modular components, separating concerns between the main and renderer processes.

Example Workflow:

Imagine building a desktop text editor with Electron:

1. Main Process:
   • Initializes the application, creates the main window, and handles file system operations (e.g., opening, saving files) using Node.js.
2. Renderer Process:
   • Renders the text editor UI using HTML/CSS/JavaScript.
   • Captures user input and sends IPC messages to the main process when the user wants to save a file.
3. Communication:
   • The renderer sends an IPC message like save-file with the file content.
   • The main process receives the message, uses Node.js to write the file to disk, and sends a confirmation back to the renderer.