SLEEC Scene Evaluation

You will be given a case study describing an agent interacting with humans and its environment, together with a set of normative rules specifying the agent's expected behaviour.

Each normative rule follows the general structure: when a triggering event or condition occurs, then the agent is expected to perform a response within a specified deadline.

In other words, each rule describes what should happen, under which circumstances, and within what time frame.

The rules presented together are related to a common context. For example, they may:

• apply under the same environmental conditions or situation (e.g., rules that apply when a patient is deaf);
• describe alternative responses that compete for the same resources (e.g., using the phone line to call emergency services versus contacting a family member);
• specify the same expected behaviour in different situations (e.g., different conditions under which the agent is expected to call emergency services).

You will then be shown a series of scenarios illustrating situations in which one or more of these rules may apply.

For each scenario, you will be asked to assess:

• Applicability: Do the rules apply to the scenario? If so, does the scenario match your interpretation of the rules?
• Rule quality: Does the scenario help you identify any limitations, ambiguities, missing information, inconsistencies, conflicts, or other issues in the current set of rules?

If you identify any issues, please explain your reasoning and describe any concerns or improvements you would suggest.

The study consists of four use cases, each containing five scenarios.

You will begin with the first use case. The screenshots below illustrate the interface and the evaluation process, and then we provide an example of the task.

Example Task

Suppose you are given the following two rules:

Notice that the two rules have different triggering events ("RobotGivesMedicationInstructions" and "RobotRemindsUserAppointment"), but they share the same environmental measure, UserIsDeaf.

You are then shown the following scenario:

Your task is to determine whether this scenario matches your interpretation of the rules.

For example, you might conclude that:

The scenario confirms my interpretation of the rules, because the robot correctly displays both the appointment reminder and the medication instructions within the required time.

However, the scenario also helps you to observe the rule set is incomplete. Although the robot displays important information to a deaf user, it never verifies that the user has actually received or understood the information. I would therefore suggest adding a new rule, for example:

When the robot displays important information to a deaf user and no acknowledgement is received within 30 seconds, then the robot must request confirmation or repeat the information.