The Admin section of ARB Suite includes Instrument Administration. It is used by the Evaluation and Assessment department to create, edit and publish assessment instruments that are used to assess medical students.

Assessments using these instruments are managed in ARB instrument admin, and completed assessments are viewed in Learning Outcomes.

Instrument Admin is a component of the ARB Suite. To access, go to ARB > Admin > Instruments.

Visual Aids

Instrument Comic

Instrument Lifecycle

Workflow

1. Give Instrument a name
2. Add and configure Instrument Elements## Drag new Elements from left panel into center section## Configure each Element using right panel while an Element is selected (yellow border) in center section## Define Element prompt
   
   1. Select Element rubric
   2. Indicate assessment validation### Required### Not Required
   3. If you want an element to not be visible when outcomes are displayed to a particular audience, or you would like to change the outcome label when the outcome is displayed, configure these here ### Visible On (defaults to visible to all, only change if outcome should be not visible to particular audience) (see Visible On, under Special Item Configurations below)#### Assessment Form#### Administrative Outcome
      
      1. Student Outcome
      2. Outcome Label (see Outcome Label, under Special Item Configurations below)
3. Save Instrument
4. Set Instrument Status (see Instrument States below)## Draft (default)## Active## Inactive
   
   1. Click Save after changing status
   2. Preview
5. Continue working later (if desired)## "Created By" and "Created When" are displayed on Instrument List view## Another person can continue work on the instrument, if desired## "Last Edited By" and "Last Edited When" is visible on Instrument List view
6. Previewing Instruments
   
   1. Instrument Administrators may preview instruments as assessors will see them, and may complete an assessment and then view how that completed assessment will look to different audiences.
   2. A preview of an instrument shows everything from the last time the instrument was Saved.
   3. Copy and past the preview URL into an email and the email recipient can click on it, view and practice-complete it (won't be saved) without being signed in.
7. Finished editing and ready to launch
8. Change Instrument Status to Active## When you're ready to begin using the Instrument for Assessments, change the status to Active
   
   1. Active means that the instrument is available for Course Coordinators to use for their assessments 
   2. When you set an instrument to Active, there is some additional validation to make sure that this instrument is ready for ~~prime time~
   3. If you've made a mistake and that instrument wasn't ready for prime time, you can change it back to Draft before it has been used for any Assessments
   4. Once the Active Instrument is saved, it will be available for use in new Assessments 
9. While an Active Instrument is in use, some basic Instrument-level information is available in Instrument Administration
   
   1. Number of Launched Assessments that use the instrument
   2. Number of times assessments have been Completed using this Instrument
10. Some cosmetic changes can be made to Active Instruments.
    
    1. You can rearrange elements within the Instrument and make edits to prompts### Changes to a single instrument will impact every Assessment that uses that Instrument
    2. You can not add new elements or edit rubrics and tags
11. Ways to retire an Instrument## Archive the Instrument### Think of Archive as a "soft" shut-off### Archived Instruments will not be available to Course Coordinators to make new Assessments
    
    1. Outstanding Assessments may still be completed by Assessors
    2. Change Instrument Status to Inactive
       
       1. Think of Inactive status is a "hard" shut-of
       2. Inactive Instruments are not available to Course Coordinators to make new Assessments
       3. Assessors can not complete outstanding assessments if the associated instrument is inactive
12. Manage all Instruments## Delete## Copy

Special Item Configurations

Populate Prompt with Definition

• This allows you to populate the prompt with the selected competency description so you do not have to write it out from scratch. (You can still edit after populating.)

Used for Grading Calculations

• Grading Reports use this to know which competency ratings to include in grading for clerkships. Historically the ratings for the four required competency domains and Overall have been used in grading, so unless there is a change in how grading formulas work, those are the ratings that should be marked as "Used for Grading Calculations."

Outcome Label

• This allows you to modify what is displayed in the Outcome for the prompt when the completed assessment is displayed, and in places like grading reports. This can be useful for making some section titles more simple in the Outcome.
• In general, these should be short prompts for comment sections. If you find yourself writing long instructions or narrative explanation, consider using Help Text or Paragraph text for that, above the comment section, and make the prompt short. If you choose not to do that, you can use this option to replace the long text with a short prompt. 
• Example:
  
  • "Please provide comments on this area" – prompt
  • "Comments" – Outcome Label

Visible On (Used for Text Fields)

• Assessment Form: This is the form the Assessor fills out. Unchecking this will hide a text element from assessors. 
  
  • This may be useful if you want to provide information to students to interpret the scores, but that is not useful for Assessors.
• Administrative Outcome: This determines what coordinators and directors can view.
• Student Outcome: This is determines what students and others view in Learning Outcomes.
  
  • This is useful for hiding help or assessor specific text and possibly the instrument title.
  • This is also useful if the student should not see a recommended grade that the Assessor fills out, but is needed for the Course Director to view for grading.

Drop-Down Response Type

• There are two different response types possible for drop-downs 
  
  • Grading Basis. This is used to allow assessors to select a recommended grade for students. 
    
    • Typically used for Electives, and hidden from the Student Outcome.
    • The grading options shown to an Assessor will be determined based on the student's enrollment in the course
  • Length of Contact shows a predetermined list of timeframes for which an Assessor may have interacted with the students. 
    
    • This field is used in some grading reports to calculate scores.

Apply Special Formatting to Text

• You can apply special formatting like bolding, italics or active links by using the buttons that appear in the Item Configuration panel. You can also use what is called markdown.
• When you highlight text and click one of those buttons, you will see that markdown such as 2 asterisks before and after bolded text will automatically be applied in the Item Configuration panel, and you will see the formatting take effect in the center preview section.
• If you're comfortable with the markdown symbols, you can also do that rather than use the buttons. 
• Using markdown, you can also apply formatting other than bold, italic and links. To view other possibilities and what the markdown for each is, read about markdown basics here. 
• Warnings:
  
  • One of the types of markdown is for bulleted list. If you type a number and a period, it will not behave the way you expect. Instead, use 1) or just 1.
  • Markdown will appear when parts of an assessment are displayed separate from whole completed assessment,  (e.g. in grading report and Learning Outcomes comment page). So use the Outcome Label field to edit what will be displayed in ARB and Learning Outcomes in place of the prompt text the assessor sees. 

Instrument States

Draft

• Instrument Administrator can edit everything for a draft instrument, including adding and removing components
• Not available for Course Coordinators to use in setting up a new assessment.

Active

• Editing is allowed while an instrument is active, until the first completed assessment has been submitted. At that point, minor editing is allowed by the system, such as examples below. But adding or removing questions is not allowed once the first assessment has been completed. 
  
  • Editing prompts 
  • Changing Competency or Competency Domains are allowed
  • Making an item required or unrequired
  • Any type of visual rearrangement
• Available for Course Coordinators to use in setting up a new assessment.
• Once an active instrument has had assignments made to it, you will not be able to change status back to draft

Inactive

• Instrument Administrator cannot edit instrument.
• Not available for Course Coordinators to use in setting up a new assessment.
• Course Coordinators cannot assign assessors for existing assessments using an inactive instrument.
• Outstanding assigned assessments cannot be completed by assessors.

Actions

Edit

• Directs you to the edit screen for an Instrument
• Editing of Instrument will be dictated by its status (Draft, Active, Inactive)

Copy

• Allows you to copy all of an instrument and create a new version
• This is useful for creating a template or Master Instrument that can then be copied and edited to meet the assessments needs for an Instrument.

Archive

• Archived Instruments are by default hidden from the Instrument list
• Editing of an Archived Instrument will be dictated by its status (Draft, Active, Inactive)
• Archived Instruments can not be selected when creating a new Assessment
• Course Coordinators can assign assessors for existing Instruments
• Outstanding Assessments may be completed by Assessors
• Archived is useful when a new version of an instrument is created. Archiving the old version will ensure that no new assessments will be created using it, while allowing existing assessments to still be completed by assessors.

Delete

• An Instrument can only be deleted if there are no Assessments associated with it.

Concern Flag/Checkbox Question Type

• The checkbox question type can be used 2 different ways, which you choose from in the Checkbox Type section of the Item Configuration panel.
  
  • Basic checkbox–just a checkbox to be used for any purpose where you want an assessor to check to indicate something
  • Used for Concern Flag–a checkbox that serves the specific purpose of allowing an assessor to call a course director's attention to a particular assessment
• When you are using the checkbox as a Concern flag there a few things you should make sure of:
  
  • On the Configuration panel, uncheck Student view of outcomes, to ensure that the assessor's answer to this question will not be visible to the student.
  • Do not make it required, as the assessor will only check the box if they have a concern
• For the prompt, type in whatever language you want to show the assessor so they know what checking the box will do, and give it a section and/or heading or not, as you see fit. 
• The Course Director or Coordinator will be able to view a report of assessments where this box was checked, by going to ARB/Reports and selecting Concern Flag for Course Leadership Review. The report consists of the full assessments for each student where an assessor checked this box.