Getting Started

Estimated time: 10-20 minutes

This guide walks you through the essential steps for getting started with Meadows, an experiment builder for online research and data annotation. Learn how to create experiments, design tasks, and recruit participants on the platform. If you get stuck, have a look at the rest of the documentation. If you feel like something is missing please contact us, we're always happy to help.

Sign up

First up, create a Meadows account. If you want to share your experiment with colleagues, we recommend creating an account with an ad-hoc shared email account (e.g. [email protected]).

Create an experiment

After logging in, you should see your user profile. This is where you'll see a list of the experiments you will design. Click Create a new experiment, and give your experiment a name that sets it apart from other projects. You could also give it a short description as an introduction for your participants later, or for other researchers if you decide to make your experiment public.

Create a task

You should now see the design page for your experiment. Click on the Tasks tab, where you can arrange the components of your experiment, adding various paradigms, forms and pages for your participants to go through. Add the first task by clicking Add a new task. Give it a name that distinguishes it from the other tasks in this experiment. Then pick the task design from the list. Every task has a question mark icon (), click this to see the documentation page for that task type. Try for example the Drag & Rate task. After you click Submit, you'll be looking at the design parameters for this task.

Select stimuli to display

Most projects on Meadows use a Stimulus, i.e. a media item that is seen or heard as part of the task, such as an image, video, sound or some text. On Meadows, a collection of these Stimuli are called a Stimulus Set. To select a Stimulus Set for your new task, navigate to the Stimuli tab. In the top section ("Source") you should see a "stimulus_set" option. Select "Google Noto". This is a public set of emoji images from Google. In the "sample" section, choose "random" and "5". On this tab you can choose other settings for the selection and presentation of stimuli. To use your own stimuli, see the section below on creating a Stimulus Set. For now press Save at the bottom of the page.

Preview: for your eyes only

This is a good moment to inspect your current design. On the right side of the page you should see the button Preview Task. Click this and you'll see your Dragrate task, as your participant would see it. In the Dragrate task each stimulus can be rated on two dimensions at the same time, by dragging each one to a location in the white rectangular arena on the left. The images you selected earlier are on the right. After dragging each stimulus into the arena, a button in the bottom toolbar lights up, to Finish the task. This takes you to a screen where you can navigate to the task results, but for now head back to the design by clicking on the name of your experiment.

Add some instructions

In order to add the necessary instructions, add a new task, and choose the General Information task type. Here you can enter instructions for the participant, potentially accompanied by a schematic or example screenshot. After entering the text and saving the parameters, head back to the experiment overview by clicking on the Tasks tab. You'd want the participant to see the instructions first, so click on an empty arena of the new task's panel, drag it up until it's at the top of the list, and release it. You've now changed the order of the tasks. Note there is a similar task called Consent Form where the participant is required to check boxes to confirm their acceptance of your ethics terms.

Create your own Stimulus Set

If you have your own stimuli that you'd like to use, you can create a custom Stimulus Set. Click on your username at the top right to go back to your dashboard. Then head to the Stimulus Sets tab. Here you should see a list of Stimulus Sets that you have access to; either because they are publicly available or because you own them. Click New Stimulus Set, and choose a name. After confirming you should arrive at a page where you can edit the new set. Under Media you can add image, audio and video files. Under Text you can add character strings. After all images have finished uploading, remember to click Save Changes at the bottom of the page to confirm your selection. If you are going to use more than a few hundred stimuli, you can instead compress the folder containing the files into a zip file, and upload this file. Once you are happy with your stimulus set, click the Finalize button to make it available for use in experiments.

Store a version

To ensure consistency in your data, you must lock your experiment design into a Version before running it with participants. Once created, a version is immutable and cannot be changed. This allows you to continue refining your design while keeping earlier participants' data tied to the exact parameters they experienced.

To create a version, head to the Versions tab of your experiment. You'll see one version listed: the special design version, which is the only version that can be edited. The current version you're viewing is also displayed in a panel next to the Preview button in the top-right.

Click Save the current version to create a new, static version. It will automatically be numbered sequentially, starting with Version 1.

Open up slots for participants

A 'slot' is a reserved space for one participant to take part in your experiment. By default, experiments have no slots, so you must create them before anyone can participate. Slots are created in Batches. We will now create a batch of slots, to enable some participants to take part in our new experiment. For this, we head to the Participants tab of the experiment version. The design version does not have this tab, so either make sure you are already browsing one of the saved versions, or go to the Versions tab and click the Recruit... link on one of the versions. On the participants tab, you'll see a button Create new batch... opening a form to configure a new batch of participant slots. Read more about this form and how to integrate with recruitment services on the recruitment page. At the bottom of the form, select the number of slots. If you create 5 slots, only five participants can take part. Note that we charge an £0.08 fee per participant minute. The first 120 minutes per researcher are free. So if your experiment takes 30 minutes, the first 4 participants are free. Please contact us to arrange payment. If you give us a heads-up when you start, we can invoice you after your experiment has finished.

Invite your participants

After creating some open slots in the previous step, the batch displays the link to share with your participants. It has a button next to it which will copy the link. You can send this URL to your participants, either through e-mail, or other methods, such as with a recruitment platform, like Amazon Mechanical Turk, or Prolific, where you can paste it into their respective forms. Find detailed guides for these platforms under recruitment page.

Keep track of participant progress and download results

At the bottom of the Participants tab you'll eventually also see a link to a page with a list of all participants. Each participant has a nickname of the form adjective-animal, for example purple-elephant. This way they have a memorable name, even when we don't collect their actual name. Click on the participant name to browse the results for this individual. You'll see a list of the tasks they participated in. Try clicking on a task, it will mention when the participant finished this task, and may include a visualization of their data. To download the results, click the download icon () which will take you to a new page with several download options. You'll also find this icon on other levels, such as on the overview for all the participant's tasks, or the list of all participants. Read more on the downloads documentation page.

Need help?

If you have any questions or run into any issues while setting up your experiment, we're here to help! Feel free to reach out to us, and we'll be happy to assist you with any aspect of using Meadows.