Psychology research using Qualtrics

Over the past month I have been developing my first memory experiment to collect data online using Qualtrics.


I was hoping to learn how to write my own code and collect data online without having to use/pay for another set of software.  However, I got some advice from a friend about how long it would take me – realistically – to get an experiment up and running that way and decided that I just didn’t have the time available to dedicate to that right now. My grant funding runs out in a few months and I need to have collected this data by then!

I currently use SurveyMonkey to collect data in questionnaire-type format, but I needed something that can present one stimulus at a time, for a set time, and record response times online.

So, I decided to try out Qualtrics research suite and I am really liking it. This first experiment has taken a while to perfect, but that is always the way. I have found the same with lab-based programmes such as EPrime – the first experiment always takes the longest to create; after that you can work from that experiment as a base and just adapt each of the pages/screens/events/stimuli to work in your new experiment.

Good things about Qualtrics:

  1. It has a user-friendly interface and it is easy and quick to add different kinds of questions (e.g., multiple choice, text descriptions)…
  2. Or to change a question from one type to another if you change your mind about something.
  3. The branching facility is also pretty simple to use – so if you want participants to see one set of blocks if they respond to a question in a particular way, or if you want participants to be randomly assigned to 1 of 3 conditions, for example….
  4. However, if you want to do both these things at once it got a little more tricky, but really it only took me about 30 minutes to figure it out!
  5. The ‘survey flow’ view is a really good feature to help with this, especially if you have participants in some conditions actually viewing some of the same blocks (e.g., if the encoding phases are different across conditions but the test phases are the same).
  6. In ‘survey flow’ you can see a list of each of the blocks that make up each of the conditions, even if some blocks are duplicated across conditions.

There are a few small things that annoy be about Qualtrics though:

  1. I can’t seem to find any decent instructions on the Qualtrics site for some of the things I want to do. I may not be looking hard enough, but the instructions I have found seem to be explaining things at a very superficial level.
  2. There are some niggly little things – like when I’ve made a piece of text bold, and it looks bold in the editing screen, but then when previewing the experiment it is not bold – arrrghhh, so annoying!
  3. There are limits to what you can change, for example, for some parts of my experiment I would like the “next” button to be in centre of the screen, or I would like the experiment to move on to show the next screen without there being a “next” button visible (you can set it to advance after a certain time limit, but not remove the “next” button) – I guess some of the things I want to do are beyond what you can change without knowing code – a reason to learn to programme myself!  Update (14/06/13) – I have now found the code and instructions regarding removing the “next” button, they could have made this much easier, but at least it can do it 🙂
  4. You can have a list (e.g., of stimuli) that are going to appear in a particular ‘field’ on a page (and the page will be repeated until all the stimuli have been shown) but you can’t copy and paste the list in from another programme (e.g., Excel) you have to physically type out each of the words. 
  5. And finally, Qualtrics is expensive! – at the moment I’m developing this first experiment using a free student account, but with that, although you get unlimited surveys – which sounds excellent, you only get 250 respondents – ACROSS surveys, so really the fact that you get unlimited surveys is a big con.

But, on the whole, I am happy with the ease of learning and using Qualtrics, and I am excited about getting my first experiment up and collecting data soon… Just have to get the ethics application approved first 🙂


11 responses to “Psychology research using Qualtrics

  1. Very helpful review. We’re trying to decide between Qualtrics and SurveyMonkey. You wrote that Qualtrics is expensive — how much is it? I didn’t see pricing on their site.

    • I don’t remember exactly… Something like $3,500 ? It varies what level you get. They will send you a very simple price breakdown if you email them. I started with a student trial account and then ‘invited my professor’ (meaning that he had to have a 1hr Skype-like session with one of their sales people) and that meant that my account got upgraded slightly. Glad you liked my review BTW 🙂

  2. Pingback: Online memory experiment #1 is ready for YOU to participate | Helen Louise Williams

  3. Pingback: New memory experiments – Participate online!! | Helen Louise Williams

  4. Hi,

    Thank you very much for your impressions in this review. You said…

    “You can have a list (e.g., of stimuli) that are going to appear in a particular ‘field’ on a page (and the page will be repeated until all the stimuli have been shown) but you can’t copy and paste the list in from another programme (e.g., Excel) you have to physically type out each of the words.”

    I can´t find the way to do that! Can you please help me with that? I am trying to present words, and i’m doing block by block! it is insane!
    Can you present words with 1 sec (or other) time in between them?

    Thank you!

    • Hi Ramon,
      I’ll try and explain…. On a new block that is going to be your ‘stimuli’ block, on the right-hand top corner click Block Options >> Loop & Merge >> Turn on Loop & Merge…. Then put your stimuli in Field 1, Field 2, etc. (if you are just showing 1 stimulus per page you’ll only need Field 1).

      Then, create a ‘Question’ in that block which is a “Text/Graphic” item type, then in the text field click Piped Text >> Loop & Merge >> Loop & Merge Fields, and put Field 1 where you want it on the page.

      You can put in a blank screen or fixation + by including a different page within this block that has a timing question on it, click New Item >> Timing >> Page Timer and setting the seconds etc. on the right hand side.

      Hope that makes sense?!


  5. That was very clearly explained! And Wow! That is time saving! Thank you very much! I’ve tested successfully.

    The only thing I still find problematic is that the words may take some seconds to upload which may depend on the bandwidth, how busy the computer is, etc. and that may interfere with the presentation of the words, as some may be in the screen for longer/shorter time. Is there any way to upload all the words, and then make the presentation? OR Can you make the word disappear right after the subject has responded?

    Thank you very much. I’ve been using Qualtrics but your post saved me many hours of inserting experimental blocks one by one!


  6. Hi Ramón, I’m glad that was helpful to you.
    You are correct that the presentation time will depend on the bandwidth etc., and there is nothing you can do about that… I tend to include timing ‘questions’ for all stimuli as these will record exactly how long each stimulus was presented for. And you can also make it that the word disappears as soon as the participant has responded – but this does involve some Java and html script so is a little more tricky.
    To make the participant’s response move the presentation on to the next stimulus see Qualtrics instructions for “Auto-Advance on Choice Selection” at the bottom of this page:

    To remove the “Next” (or “>>”) buttons on the stimuli screens put this HTML code into the actual text box (i.e., box where stimuli field is). This goes below whatever text is already in there:

    .Skin #Buttons {

    Hope that is useful to you.
    P.S. Sorry for my delay in replying, I was out of the country.

  7. Hi Helen,

    I’ve just come across your blog and finding it very useful. I’m new to Qualtrics but wondering if you’d be able to help me out with something please?

    I’d like to show a prime but it looks like the shortest duration is 1 second, which I think may be too long (even with supraliminal priming?). Do you know if prime presentations can also be displayed in ms as RTs can?

    Many thanks

    • Hi Victoria, sorry for the delay in replying! Unfortunately I don’t know the answer to your question as I’ve never needed to present a prime for less than 1 second. I suggest asking the Qualtrics support people, I’ve always found them to be really helpful. They could probably help you to write some script to present stimuli for short time intervals. Good luck with your research! Helen

      • Thanks Helen! No worries, I decided to add the short presentations as a YouTube video, which seems to have worked OK:)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s