1. How do I find resources on filtering?

• Go to Learn More about Filtering in Displayr

2. My filter has become invalid

• If you receive a red QFilter error then this signifies that there is an issue with the filter applied to your output:
  
• You should find the filter under Data Sets, click it, and then update the condition where necessary.
  
• If the error is in a variable that your filter uses then it will indicate the invalid variable that requires fixing.
• If the issue has been resolved but the output still shows as invalid, untick and tick Automatic.

3. Applying a filter to a visualization produces a warning

• If you receive an orange warning like the below, this is because any filters should be applied to the source table rather than the visualization:
• It will not affect your results but ensure the source table has the correct filter applied.

4. Applying a filter to my output produces a 0 observations error

• If your table returns the below error but you expect there to be data, you should check the condition of the filter variable.
  
• Check the raw data via Insert > More > Tables > Raw Data by selecting the filter variable and the source variables used in the condition. This will allow you to see if the issue is the actual data or simply the filter.

5. Applying multiple filters to my output doesn’t produce the desired result

• Check the logic conditions in the individual filter variables.
• Check the raw data via Insert > More > Tables > Raw Data by selecting the filter variables and the source variables used in the condition.
• Check that the filter variables are grouped correctly. Non-grouped filters are treated as an AND condition whereas grouped filters are treated as an OR condition.
  • If your filters should be grouped (i.e. perform an OR condition), select them under Data Sets and go to Insert > Data Manipulation > Combine.
  • If your filters should not be grouped (i.e. perform an AND condition), select them under Data Sets and go to Insert > Data Manipulation > Split.

6. My R output doesn’t filter even though a filter is applied

• If you have an R output with custom code that does not reference QFilter and then apply a filter, you will get the following warning:
  
• QFilter is essentially a reference that stores the record-level TRUE/FALSE generated from the filter(s) selected under Input > FILTERS & WEIGHT > Filter(s). See Filters in Displayr for further details.
• If you are dealing with a single variable, adding [QFilter] to your R reference will then apply the filter(s) selected on your output:
  
• If you are dealing with multiple columns of data then [QFilter,, drop = FALSE] is more appropriate as the filter needs to specifically be applied to the rows while leaving the columns as is.

7. Unwanted filters are showing in View mode

• By default, all filters in your document are visible in your published dashboard. Any filters you don't want to show can be hidden via the Hidden except in the data tree option:
• Note that you cannot hide filters that are currently in use in an output. You must do this prior to applying the filters.
• It is possible, however, for Q users to open their Displayr document QPack and set the filters hidden in the Variables and Questions tab using the H in the Tags column. Please use the workflow outlined in How to Edit a Displayr Document in Q.

8. I've opened my Displayr document QPack in Q and it's asking me if I want to reset my filters

• While you can open Displayr documents in Q, filters that are created in Displayr work differently. In Displayr it is possible to group filters to allow you to join filter variables with OR. Q, by contrast, can only join filter variables with AND. If you select an output where such a filter exists, you should select No to maintain the current set up.
• See the Combining filters section of How to Filter Data in Displayr for more details.

9. I'm trying to create a filter using R code but it doesn't work

• Go to How to Use Basic R Code for Creating Filters

10. I have unwanted items in my control box list

• Items to be displayed come from both the Item list and Items from fields. By default, First option; Second option will appear under Item list but you can remove these. If you, therefore, have overlapping items listed between the two options, you will need to remove one of these duplicates:
  
• If you are only using the Items from option, you can Hide items from the referenced table or amend your R code accordingly.

11. My output(s) won't update when connected to my control box(es)

• Check Automatic is ticked on the output
• If you are using manipulation settings: Check the correct control box has been referenced under ROW/COLUMN MANIPULATIONS > Rows/Columns to show and Select rows to show by is set to Choosing from Combo Box or List Box control.
  
  Remember, if you have used the DATA MANIPULATION > Switch rows and columns option, you will need to apply these settings in the reverse manipulation menu.
• If you are using filters: Check that you have applied the correct filter to the output and ensure any source table is included on the same page.
• If you are using multiple filter variables, ensure they are all applied to your source outputs.
• Check that the filter condition references the correct control box name.
  
  If using a Page Master, you will need to first find the correct template via Appearance > Page Master. Remember if you copy all the outputs from one page to another, the reference names will be different.
• Check that the control box items match the source question being referenced. Remember that any merging of table categories for the source question will also need to be reflected in the control item list.
• Check that you are using the correct formula as single and multiple-response questions require different code. See How to Connect Filters to a Combo Box (Control) for further details.
• Below is an example of a single-response condition based on the categories from Age: 

Age %in% Combo.box

• And this is an example of a multiple-response condition based on the variables from Rolling months:

rowSums(`Rolling months`[, Combo.box, drop = FALSE]) > 0

• If you are using a filter variable that combines all the conditions, ensure they are all connected with the & (AND) symbol. For example:

Age %in% combo.box.age & Gender %in% combo.box.gender

• If you are using a single-response question as the basis of your control categories but have created overlapping NET categories, you will need to set this up instead as a set of binary variables. You can easily do this via Insert > Filter > Filters from Selected Data. You will then need to adjust the filter formula to reference these new filter variables.

12. I added a total sample item to my control box but my output(s) won't show total base

• Check the formula condition in your control filter variable. Filter variables need to return 1 or TRUE when selecting records and 0 or FALSE when not selecting them.
• The general way for setting a 'total sample' default among custom filter categories is by adding an if/else statement. Below is my existing filter condition:

usergroups %in% Combo.box.groups

• I also have an item in my combo box called Total Sample so I need to add that condition and wrap it all within an if/else statement. To do this I can use the rep function and reference the length of the id variable which signifies the number of records in the file. This will then generate a 1 for every record in the data set if this condition is TRUE, otherwise, it will return the second filter condition.

if ("Total Sample" %in% Combo.box.groups) {
rep(TRUE, length(id))
} else {
usergroups %in% Combo.box.groups
}

• If 'total sample' is instead based on another filter variable, check its logic condition.
• Check any applicable raw data via Insert > More > Tables > Raw Data by selecting the filter variable and the source variables used in the condition.

13. I've applied a filter in View mode but it is no longer applied when I go to another page

• View mode filters only apply to the current page. See Allowing Users to Filter Pages in Dashboards and Using Displayr to Filter Data, Analyses, and Whole Reports for further details.
• If you wish to apply global filters to your document, you have 2 choices:
  • Create one or more control boxes on a single page whereby every subsequent page has a filter applied that references these controls.
  • Add your control box(es) to a Page Master and ensure this layout is used on every page you wish to apply this filter. This will result in any control user selections on one of these pages being reflected in all the other pages using this layout. See How to Use the Same Control on Multiple Pages.

14. I'm trying to export reports with selected filters but my reports aren't filtering properly

• The Select multiple filters to export multiple reports option requires all underlying tables and outputs to be on visible pages. Any items on hidden pages will not be filtered. It works in the same way as applying page filters from the Filters drop-down menu in View mode.
• If you are using tables as the underlying data, you should place them on the same pages as your final outputs.
• If you are using R outputs as the underlying data, these also need to be filterable, that is, they need to reference QFilter in the code. Unless you are using a custom R output, this will already be included in the underlying R code.
• See How to Automatically Export Multiple Reports with Different Filters for further details.

15. I'm trying to display the sample size description but it won't show the filter label based on the control box

• When filtering data through a control box, you may wish to display the sample size description via Insert > More > Data > Sample Size Description. In this case, the underlying code behind the widget won't be able to deal with this by default. To customize your code, go to the Advanced Customization: Dynamic updating with Combo or List Boxes section of How to Customize the Sample Size Description Widget.

Creating binary filters from selected data

When creating a single filter variable, you can simply go to Insert > Filter > New Filter and choose the appropriate category. However, repeating this step for each category can potentially be time-consuming. Instead, creating many filters quickly is as easy as selecting the relevant variables and going to Insert > Filter > Filters from Selected Data. This will produce a filter for each category in the source question. Note that if you merge any categories via the related table, they will instead flow through to these filters, and any that have been removed will be excluded.

The grouped filters will appear below the source question under Data Sets.

A second use for this function is when you have Binary - Multi (Compact) or 'Max-Multi' questions that you wish to convert to binary data. This will in turn produce a single variable for each category. You can then untick the Usable as a filter option in the output if this is not needed.

Creating combo and list box controls and filters

While it generally takes a few steps to connect filters to a control box, there are shortcut options available.

In this example, I have the following grid of bars chart showing preferred cola over time:

To automatically apply a control filter, you should do the following:

1. Select the output(s) and go to Insert > Filter > Combo Box (Drop-Down) Filters on an Output or List Box Filters on an Output. We will choose a list box.
2. Next, choose the single-response variables you wish to use for the filter. Here I have chosen 'D1 – Age' and 'D3 – Gender'. Note that for each question selected, a separate control box will be created.
3. If you are creating combo boxes, you will need to additionally select Yes to allow the user to select more than one category in each control box.
4. A new combined filter variable called 'List Box Filter D1 – Age + D3 – Gender' has now appeared under Data Sets:
   
5. 2 list boxes also appear on the page with the filter automatically applied to the chart:
   

Creating text box controls and filters

An alternative to a combo or list box is a text box filter. You can use this for filtering lists of verbatims or any other output for that matter. The process is almost the same via Insert > Filter > Text Box Filters on an Output except for 2 steps:

• In step 2 above you will only have the choice to select text variables for creating the filter.
• In step 3 you will be asked if you want the filter to ignore case, that is, it doesn't matter whether you use capital letters or not.

The result will be a text field that you can type into to filter the other outputs. In the below example, we have based our text filter on an open-ended cola awareness question. When we then type "cola" into the field, it automatically filters the text question and the chart based on those who wrote "cola" in their verbatim response.

Rebasing multiple-response data to the NET

There are often situations when missing value settings in your data set are incorrect for multiple-response questions due to respondents skipping the question. This can lead to the NET not being 100%.

In the below example, there are 11 records out of 725 which have no data so we want to rebase the question out of those who actually answered it. This can be easily rectified via Insert > Utilities > Create New Variables > Rebase Multiple Response Data in Variable(s) to NET. This option will create a new rebased question at the bottom of your Pages section and provide the option of hiding the incorrect version in the prompt.

Rebasing questions based on other questions

Data exports can often have limitations with how missing values can be set when questions are filtered within the survey. This can leave the data file with incorrectly based questions. Alternatively, you may simply want to rebase your data differently on awareness or usage, for example.

In the following example, we have a multiple-response grid, Q14, where the respondent was asked to identify what supermarket brands correspond to specific attributes.

The table is currently based out of total sample but we would like to base it out of Q8, that is, whether they have shopped at these stores in the past 12 months:

The steps to follow are below:

1. Go to Insert > Filter > Filter One Question by Another Question
2. When prompted, select the question you want to filter then press OK. We will select Q14 here.
3. When prompted, select the question to use as a filter then press OK. Here we will select Q8.
4. If the question to be used as a filter is tagged with Usable as a filter, you will be asked whether you wish to simply apply this filter (Yes) or create a variable for each category (No).
5. If the question to be filtered has multiple variables, you may be asked whether you wish to split each variable by the filter categories (Yes) or simply apply the matching filter (No).
6. If you want Displayr to automatically match up the filters based on the source labels, select Yes.
7. As our labels match perfectly, Displayr will generate a new version of our question with the correct basing called 'Q14 filtered by Q8 Visited last 12 months':

The table looks like this:

6. Note that if you select No in step 4 or there are inconsistencies with labels and/or additional items, you will need to manually select the variables that correspond to each filter category via the prompts.
7. You can then choose whether to keep or remove the extra items that were not matched.

To avoid these extra steps, you can simply adjust the variable labels under Data Sets to provide consistency. If the difference, however, is only that the filter question also includes a "None of these" option, the easier solution is to copy this question via Home > Duplicate, go to DATA VALUES > Values, and set this option to Exclude from analyses. Now you can use this question instead and the items will match.

Note that, much like with the Filters from Selected Data option, here Displayr will also respect any merged or removed categories when creating the new variables.

Filtering variables

A second use case for Filter One Question by Another Question is for filtering your variables to allow for a 3-dimensional table.

Below is a table of how likely respondents are to recommend buying fruit and vegetables as an average crossed by the last supermarket they visited.

Although these 2 questions are easily crossed, you may still want to cross break the table further. This is where this script comes in handy.

As with the first few previous steps, you simply need to do the following:

1. Go to Insert > Filter > Filter One Question by Another Question
2. When prompted, select the question you want to filter then press OK. We will select 'Q12 Fruit & Veg'.
3. When prompted, select the question to use as a filter then press OK. Here we will select 'Q10 Last visited supermarket'.

The result is a simplified table which can now be further filtered by applying a banner question, such as gender:

Splitting sample

Displayr also offers you an automated solution for splitting your sample for predictive modeling such as regression.

The first option via Insert > Filter > Filters for Train-Test Split randomly creates a training and a testing filter. This defaults to a 70% / 30% split but you can change this in the prompt.

The second option via Insert > Filter > Filters for Train-Validation-Test Split randomly creates a training, validation, and testing filter. This defaults to a 50% / 25% / 25% split but you can likewise change this in the prompt.

Another use for this function is for effectively removing additional records from your data set if you have gone over your survey targets. In this case, you could use the Filters for Train-Test Split option and set the training sample accordingly.

If you wanted to remove 5% then you could use either 95 or 5 for the prompt and choose the appropriate filter:

This results in 2 grouped filters at the top of your data set:

At first glance some of the R code may look scary to those of you who have never used it before but it offers a lot of flexibility. We have already covered how to create and connect filters to a control box. The following deals with other common scenarios that can be easily updated to fit your requirements.

The process to create an R filter is to go to Insert > R > Numeric Variable, add the condition in the R CODE window and then select Usable as a filter.

The R Basics

Before we begin, let's cover some variable and question referencing and condition operators.

With R you can reference either the variable name or the question name/label. If the question name has spaces in it, however, you must wrap it in back ticks. As an example, I have a single variable called Q1 with the label of "Q1. How old are you?". To reference the variable, I would use Q1 but to reference the question it would be `Q1. How old are you?`. In this case, using either option produces the same result but the distinction can be important as a question could be made up of multiple variables.

The dimensions of a question are question[rows, columns, drop=False] whereby you may choose to specify or exclude specific columns from being included in the columns argument. In order to keep the original dimensions, we add drop=False or drop=F as an additional argument.

If you are referencing a single column, you can do so using the column name, e.g. "Coca-Cola". You can also use column index but if you change the order of items in the corresponding table of this question, this number may change. If you are referencing multiple columns then wrap them in c() such as in c("Coca-Cola","Pepsi").

The main condition operators are as follows:

Filters based on single-response questions

When dealing with single variable questions, whether numeric or categorical, it is easy to set up basic binary 'Yes' or 'No' conditions.

1. 'Yes' Filter

Q1 is 1:

Q1==1

Q1 is less than 5 OR Q2 is 2:

Q1<5 | Q2==2

Q1 is greater than or equal to 25 AND less than or equal to 100:

Q1>=25 & Q1<=100

2. 'No' filter

And here are the reverse of the above filter conditions using the ! operator.

Q1 is NOT 1:

Q1!=1

Q1 is NOT less than 5 AND Q2 is NOT 2:

!(Q1<5 | Q2==2)

Q1 is NOT greater than or equal to 25 AND NOT less than or equal to 100:

!(Q1>=25 & Q1<=100)

Filters based on multiple-response questions

Using R to create filters based on Binary – Multi questions has the advantage of being able to reference a 'question' rather than simply individual variables. This is essentially the same as viewing your data as a table with each variable as a column and the rows as the respondents. When you have a long list of statements, you can therefore get away with specifying the least amount of columns for your condition to work.

The below uses the example of a binary question called 'Awareness - Colas' where each cola is a different variable.

1. 'Only' filter

Only aware of Coca-Cola:

q = `Awareness - Colas`
rowSums(q[,"Coca-Cola",drop=F])>0 & rowSums(q[,!colnames(q) %in% c("Coca-Cola","NET"),drop=F])==0

Here, we are counting anyone who selected Coca-Cola and then summing per row any column that is not named 'Coca-Cola' or 'NET' and returning any that were not selected. In this case, it is easier to define the question as 'q' first when using it multiple times in the code.

Alternatively, you could replace rowSums(q[,"Coca-Cola",drop=F])>0 with the name of the variable, e.g. q2_1.

2. 'Count' filter

Aware of more than 1 cola:

q = `Awareness - Colas`
rowSums(q[,!colnames(q) %in% c("NET"),drop=F])>1

Here, we are summing all the columns per row (excluding NET) and returning any that selected more than one. You can in turn change the value at the end to match the total number of items for creating an 'All' filter.

3. 'Any' filter

Aware of any colas:

rowSums(`Awareness - Colas`)>0

Here, we are summing all the columns per row and returning any that were selected.

4. 'None' filter

Aware of no colas:

 
rowSums(`Awareness - Colas`)==0 

Here, we are summing all the columns per row and returning any that were not selected.

Filters based on dates

Date formats can vary but half the battle is won if you are using a standard date format. The below examples assume you have a Date/Time variable in your data set. If not, you will need to convert the date first. This may be as simple as changing the Structure to Date/Time, provided it is already in a recognizable format. Otherwise, you can convert strings to dates and adjust timezone, if necessary.

The below uses a date variable called 'StartDate'.

1. Before/After Date

'StartDate' is before October 2020:

 
StartDate < "2020/10/01"

Here, the date format must match that of 'StartDate', i.e. YYYY/MM/DD. If you are using a date control (via Insert > Control > Date) that selects the date, you can replace "2020/10/01" with the name of your date control.

2. Date Range

'StartDate' is between October and December 2020:

 
StartDate >= "2020/10/01" & StartDate <= "2020/12/31"

Again, the date format must match that of 'StartDate', i.e. YYYY/MM/DD.

3. Last/Next K period

'StartDate' is within the last 3 months:

library(lubridate)
StartDate >= Sys.Date() - months(3)

Here, the 'months' argument is from the lubridate package so we need to reference this library. This can be changed to days, weeks or years, and you can change the – to a + for future periods. 'Sys.Date()' is simply the system date or the current date on the server.

4. Previous/Current/Next period

'StartDate' falls within the current quarter:

library(flipTime)
Period(StartDate, "quarter") == Period(Sys.Date(), "quarter")

Here, the 'Period' argument is from our flipTime package so we need to reference this library. The quarter as defined by 'StartDate' is then compared to the current quarter based on today's date. The 'quarter' argument can be changed to day, week, month or year. You can also add the period arguments from the previous example to adjust when the period falls. For example, comparing 'StartDate' to next quarter is as simple as adding 3 months to today's date:

library(flipTime)
library(lubridate)
Period(StartDate, "quarter") == Period(Sys.Date() + months(3), "quarter")

This allows you to choose what table dimensions should be included in the final output without having to make changes to the source table. If you wish to display a table instead, referencing your source table via a Paste Table (Autofit) R output provides the same manipulation options.

Let's now go through the different types of filtering possible under the Inputs > ROW MANIPULATIONS and COLUMN MANIPULATIONS sections of these outputs.

Remove rows and columns with no data or small sample size

1. Hide empty rows/columns
This option removes any row or column that has no data in it.

2. Hide rows/columns with small sample sizes
Instead of only removing blank data, you can also specify a Sample size cut-off to remove rows and columns that do not meet this threshold. Note that the statistic used for sample size will be based on your source table, so Column Sample Size, Column n, Sample Size, or Base n must be included.

Display fixed rows and columns

3. Select rows/columns to show by > Type row names or indices
If you wish to show only specific rows/columns then you can list either the name or index number of the desired row/column under Rows/Columns to show, separating each item by a comma.

4. Rows/Columns to ignore
By default, NET, Total, and SUM are set to be ignored from your output but you can change this or add to the list by referencing the name of the row/column, again separating each item by a comma.

Display dynamically selected and auto-updating rows and columns

5. Select rows/columns to show by > Choosing from Combo Box or List Box control
When you have numerous brands or statements in a table, you often want to let the viewer select the items they are interested in via a Combo Box or List Box. This option will work seamlessly with any visualization or pasted table output when you select the appropriate control box under Rows/Columns to show. The result will be an output that automatically changes what data is displayed based on the user selections.

6. Number of rows/columns from top/left to show
If you have additional rows/columns in your source table and only want to show the first 6, for example, then setting this to 6 will always show these rows/columns even if your data updates. In this example, only 'Coca-Cola' to 'Pepsi Max' have been included from the table rows.

7. Number of rows/columns from bottom/right to show
This option is the same as the above but works the opposite way. If you have quarterly time series data and you want to only show the last 2 quarters i.e. the last 2 columns of your table, setting this to 2 will restrict the output to only show these columns. If your data then updates with additional months, this option will automatically adjust.

What is Data Filtering?

What is Rebasing?

5 Things to Consider Before Building Your Dashboard

Variables and Questions

5 Ways to Create a Filter in Displayr

How to Filter Data in Displayr

How to Use Basic R Code for Creating Filters

Easy Functions for Automating Filters and Rebasing

Filters in Displayr

Managing Filters and Weights

10 Ways to Create New Variables in Displayr

Using Support Vector Machines in Displayr

Control boxes

Adding a Combo Box to a Displayr Dashboard

How to Connect Filters to a Combo Box (Control)

Combo Boxes (Controls) With Dynamic Lists in Displayr

How to Use the Same Control on Multiple Pages

How to Dynamically Change a Question Based on a Control Box

Using Controls to Determine when to Show Outputs

How to Switch Logos and Images Based on User Selections

How to Customize the Sample Size Description Widget

Optimizing your Conjoint Analysis Simulator in Displayr

R Visualizations, Tables and Outputs

How to Filter Rows and Columns in Visualizations and Tables without Code

How to Remove a Row or Column using R in Displayr

Filtering a Subset of Tables and Visualizations on a Page in Displayr

Creating tables with multiple variables (filters and multiway tables)

Pages and Documents

Using Displayr to Filter Data, Analyses, and Whole Reports

How to Filter a Dashboard Based on User Logins

Allowing Users to Filter Pages in Dashboards

How to Automatically Export Multiple Reports with Different Filters

Troubleshooting Guide, Videos, Tutorials and Dashboards

Troubleshooting Guide and FAQ on Filtering

Using R in Displayr Video Series

Tutorial: Filtering Data

Dashboard Design: 8 Types of Online Dashboards

Dashboard Example Gallery

In the example below, the combo box says Hide Comments. If you click on it and change it to Show Comments a comment appears.

How it works

There are three stages to doing this:

1. Adding a control to the page
2. Creating the content you wish to show.
3. Hooking the content up to the control.

1. Adding a control to the page

In the example above, I've used a combo box: Insert > Control > Combo Box (this option is on the very far right of the Insert tab in the ribbon at the top of Displayr). In the object inspector on the right-side of the screen, enter the options that the user can choose from in Control > Item list. In this example, I used Hide Comments; Show Comments.

2. Creating the content you wish to show

This is the hard bit. We need to create a table or something that is written in R code. So, if we want to create a textbox, for example, we need to do it in R code. The example below is created using Insert > R Ouptut and then adding in the code.

The text is shown in line 1. The rest of the code does the formatting, using HTML.

You will probably want to hide this content (Home > Hide)

3. Hooking the content up to the control

• Insert > R Output
• Paste in: if (Combo.box == "Hide Comments") "" else my.comment
  If necessary, change Combo.box and my.comment to whatever names you've given these objects.

Now, just use the control and you will see that the comments appear and are then hidden, depending on what you have selected.

Adding lots of conditional outputs

If we need to have lots of outputs that appear based on controls, it could have some performance implications. There are a few things you can do. One is to combine steps 2 and 3 together, as shown below. Another option is to have one R output that contains lots of different things. For example, all the text boxes could be created in a single R Output if you have the HTML skill (this is a complicated thing to do; only investigate it if you have a performance issue due to too many R Outputs on the page).

The process is quite simple:

1. Create filters in your document
2. Publish the document to the web
3. Export multiple documents using the filters

Create your filters for the loop

There are many ways to create a filter in Displayr - see: 5 Ways to Create a Filter in Displayr for more detail. Importantly, if you create filters using Method 4 then you will need to set the variables together in a Binary-Multi variable set. To do so, select all variables in the Data Set that should be set together, and go to Data Manipulation > Combine from the Ribbon.

Publish your document to the web

Once your filters and report are ready, publish the document to View mode. To do this, go to Export > Web page. Remember to Republish if you have previously exported.

Export to PowerPoint looping through each of your filters

Within View mode, click on the Export link (at the very top). Select the file type to export: PowerPoint, Excel, or PDF.

Then, click on Advanced Options, and pick the filter categories you made above and click Export. Displayr will generate one report file for each selected filter category!

To avoid interrupting the export process, please keep the browser window open until all the exports have downloaded.

Watch the video below to see the looped exports in action, or try it for yourself using our example Campground Dashboard.

For example, the image below shows a question (Preferred Cola) that I've chosen to split by income brackets, using the selection in the control box.

If I change the control box option to Age, it becomes:

You can do this with an R variable. The R variable dynamically updates when the selection in the control box changes. The purpose of this post is to show, via example, how you can do this.

Setup your control box with your options

Use Insert > Control and then choose either a Combo or List box. Over in the Object Inspector, list your questions in CONTROL > Item List (which can be labeled however you like). In this example, I entered 4 possible options for a combo box:

I set the Selection mode to be "Single selection," and When item list changes to be "Select first."

Be sure to take note of the control box’s name under PROPERTIES > GENERAL > Name, because we’re about to use this in the R variable.

Changing single-variable questions via your control box

Next, you will need to create an R variable with conditional statements that link to the questions via Insert > R > Numeric variable. This will make a new numeric variable under Data Sets, creatively called “newvariable” by default. Displayr will reveal in the Object Inspector a blank box where you can put in the R CODE:

As per the picture above, you enter simple conditional statements with R. Basically, it references the control box (called Combo.box in this example) and then each of the 4 options. The four variable names -- d1, d2, d3, and d4 -- pertain to each of the single-variable questions to use in the table. The code consists of very straightforward "IF and ELSE IF" statements.

Be sure to change the variable Structure to be nominal or ordinal (if you intend for the question to be categorical). This is done under INPUTS > Structure in the Object Inspector for the R variable (in the picture above at the very bottom under the code).

And that’s it! From there you can use your R variable in a table, directly in a visualization, or in another analysis. It will change dynamically as you alter the selection in the control box.

Changing multiple-variable questions via your control box

When working with multiple-variable questions, it may be possible to use the same approach of using 'if/else' code for each variable in your variable set, but there are some provisos:

• Your variables must be set together as either a Binary – Multi or Number – Multi, as applicable.
• You should have the same number of variables for the questions that are to be substituted.
• The variable labels should be applicable for all questions, as these can't dynamically change.

When the number of variables and/or variable labels are different between the questions you wish to dynamically change via a control box, it is better to substitute tables instead. The steps are as follows:

• Create separate tables for each of the questions listed in your control box, drag them off your page and select Appearance > Hide from the ribbon.
• Create an R output via Insert > R Output that selects which table to choose based on the table name (found under PROPERTIES > GENERAL > Name) and the control box selection:

if (Combo.box == "Awareness") table.D1.Age.by.Awareness else
if (Combo.box == "Preference") table.D1.Age.by.Preferred.cola

• In the above example I have 2 control options that switch between 2 tables, one 'Age by Awareness', the other 'Age by Preferred Cola'. As the final output is a visualization, I've also hidden this R output and dragged it off the page.
• Once you update the visualization's output reference under Inputs > DATA SOURCE > Outputs in 'Pages' to this R output, you will then be able to dynamically control the data shown:

Changing the weighting dynamically with an R variable

You can apply the same technique to dynamically change the weighting. You essentially reference different weighting variables in the R code based on your selection in the control. For example:

if (Combo.box == "USA") weight_us else
if (Combo.box == "France") weight_fr else
if (Combo.box == "UK") weight_uk

Then make sure the R variable has the Usable as weight box checked in the Object Inspector. You can then apply that to a table (or chart or whatever) as your weighting variable.

Try for yourself

Worked example

In this example I start with a table, called country.lookup, which contains countries by continent. This was created by selecting Insert > Paste Table, pasting in a table, and changing its name in Properties > GENERAL > Name to country.lookup.

Creating a control from an R Output

An R Output is an arbitrary calculation in Displayr. A control is created from an R Output as follows:

1. Create an R Output that contains a character vector of the options for the control. In this example, this is done by
   • Clicking Insert > R Output
   • Typing continents = unique(country.lookup$Continent), which extracts all the unique continents from the Continent column of the table.
   • Pressing  CALCULATE  
2. Insert > Control (More) and set:
   • Properties > GENERAL > Name to cContinent
   • Control > Items from R Output to continents
   • Delete entries in Control > Item list
3. Click on the control and select any country (the rest of this post assumes you selected Asia)

Creating a control that is linked to the first control

Now we will create a second control, the contents of which are linked to the user's selections in the first control. This basic approach can be extended to much more complicated examples, allowing users to drill into large databases.

• Click Insert > R Output and insert the following code, which selects all the countries int the selected continent:

    countries = country.lookup$Country[country.lookup$Continent %in% cContinent]

• Insert > Control (More) and set:
  • Properties > GENERAL > Name to cCountry
  • Control > Items from R Output to countries
  • Delete entries in Control > Item list
  • (Optionally) Set Selection mode to Multiple Selection
• Click on the control

If everything has gone to plan, your screen will now look like this (if you do this without having published using Export, you will get some little circles on either side of the control).

Check out more handy tips on Using Displayr!

Click here for an interactive tutorial on interactive controls

• Look at results for a particular period of time.
• Calculate results for particular groups of interest.
• Exclude erroneous or "bad" observations from an analysis.
• Train and validate statistical models.

Filtering requires you to specify a rule or logic to identify the cases you want to included in your analysis. Filtering can also be referred to as “subsetting” data, or a data “drill-down”. In this article we illustrate a filtered data set and discuss how you might use filtering.

Example of filtering

The table below shows some of the rows of a data set from a survey about peoples’ preferred Cola. The survey data contains demographic information about the respondents as well as each person’s preferred cola and that person’s rating (out of 5) for each of six varieties of cola.

Filtering this data involves:

1. Coming up with a rule for the observations needed.
2. Selecting the observations that fit the rule.
3. Conducting the analysis using only the information contained in those selected observations.

For example, the table below show the data filtered for Males only. The darker colored rows are kept in the analysis while the remaining rows are excluded. Results computed for Males are then calculated based on the highlighted rows (ID’s 2, 9, 11, 12, 13, 14). If we want to know the average rating for Coca-Cola among males, we would compute that as (5 + 5 + 4 + 5 + 5 + 3) / 6 = 4.5.

Results for different groups

A basic need for most research is to obtain results for different groups in the data. One may want to ask about the prevalence of a disease within a demographic segment of the overall population, understand sales figures for the past 3 months, or view feedback given by customers who gave your restaurant 1 star on Yelp. In each case, a logical rule defines whether each case in the sample is excluded or included.

From the example above, we may wish to compute the average rating for each beverage within for the Males in the sample. Such filtering transforms the results like this:

Sometimes filtering is carried out implicitly. For example, in survey research, the columns of a crosstab correspond to a special case of filtering, where filtered results are computed separately for each column, and the results are displayed side-by-side.

Data cleaning

One reason for filtering data is to remove observations that may contain errors or are undesirable for analysis. For example, you may want to remove respondents who did not complete the survey, respondents who raced through the survey and selected answers without paying attention to what they were answering (“speeders”), or cases where data entered manually has been entered with mistakes. In other areas of research, a multivariate technique may only be applicable to cases where there is complete information for all the variables that were measured, and so a filter may be constructed to remove cases where some observations are missing.

Checking results

Filtering can be used to evaluate the performance of statistical algorithms and models. The basic idea is to split up the sample into two or more groups, and to then apply the analysis independently to each group and compare the results. This kind of filtering would select cases from the data at random, rather than using some rule which is based on the data. This ensures a valid comparison and is often referred to as training, testing, and validating.

Data filtering by software

Sign up for Displayr

Step 1: Import your data

Start by opening a document that contains a Data Set. For this example, I used demonstration data from a brand tracking study.

Step 2: Create a table to be filtered

Add whatever you wish to show on the page. I have dragged the Age variable across from the Data Sets panel to form a table.

Step 3: Create the control

Step 4: Set the control properties

Step 5: Creating the filter

Go to Insert > R > Numeric Variable. Make the following changes: Set the Label to Age filter Check Usable as a filter Set the R CODE to Age %in% Combo.box. If you are using your own dataset, replace Age with the name of the variable you want to use to filter. If you have changed the name of your control, or have any existing controls, you will need to replace Combo.box with the correct name. If you are filtering with multiple response data, the R code is: rowSums(variable.set.name[, Combo.box, drop = FALSE]) > 0 The following are required in order for this to work: The multiple response data (e.g., brands consumed in the past six months) has a separate variable for each alternative in the combo box. The labels are exactly aligned. the variable set is of structure Binary - Multi.

Step 6: Apply the filter to the items on the page

Select the item that you wish to filter, and choose the filter you have just created in Object Inspector > Inputs > FILTERS & WEIGHT > Filter(s). The example below applies the Age filter to a bar chart, allowing users to filter data using the combo box. Alter the combo box selection to alter the bar chart.

Step 7: Export your dashboard

Now go to Export > Web Page. You can now select categories from the combo box and the outputs and the page will automatically update.

An Alternative Solution to Filtering a Visualization

If you are specifically looking to filter the rows/columns from a table to show in a visualization, you can skip steps 5 and 6 above and use your control box directly in your visualization. From the Object Inspector of your visualization, you can filter the rows by changing Inputs > ROW MANIPULATIONS > Select rows to show by to Choosing from Combo Box or List Box control and selecting the name of your control from the Rows to show drop down. There are corresponding fields to filter columns under Inputs > COLUMN MANIPULATIONS. 

Click here for an interactive tutorial on interactive controls

The user experience

When a user accesses Displayr in View Mode, the top-right of the screen shows a standard set of controls.  These allow searching, filtering, exporting, zooming, and getting a full screen display. The image below shows these controls.

When you click Filters it shows a list of set-up and available filters in the project. In this example, the user's selections equate to Males aged 18 to 34 (i.e., Male AND (18-24 OR 25-29 OR 30-34)).

Please click here to access an example of a Displayr document where the user can filter analyses and visualizations.

Creating filters

There are a number of different ways to create filters in Displayr (see 5 Ways to Create a Filter in Displayr for an overview). When creating filters for use by a user in Edit Mode, the best method is usually to:

• Select the variable sets you wish to use to create the filters from the Data Tree (bottom-left of the screen).
• Insert > Filter > Filters from Selected Data.

When you do this, related filters group together as variable sets. This is important because when the user in View Mode selects Filters they will see the filters in the same variable set grouped together, as shown above. Furthermore, filters within the same group work as OR operations.

Ensuring the filters are hooked up to the data

Any tables created in Displayr by dragging or dropping, or created in Q and imported into Displayr, will automatically be filtered if the the user selects from Filters and presses OK. Similarly, charts created from a table using Home > Chart will also have their contents automatically filtered.

If using an R Output, and this includes all the charts and other visualizations in Insert > Visualization, the filtering needs to be set up in one of two ways:

• Have a Displayr table on the same page (presumably hidden from view using Appearance > Hide), with the data in the visualization linked to this table. Thus, when you update the table, the R Outputs linked to it will automatically update.
• By explicitly addressing the filter in the R code in the R output (using the variable called QFilter).

Click here to sign-in to Displayr and access a document that addresses filters in R code.

Click here for more information about applying filters to Displayr documents.

This post describes four strategies for restricting the filtering so that it only applies to a subset of the items. Strategies 1 and 2 are to create outputs that do not interact with the page filters. Such items will not update when a user changes the filters. Strategy 3 is to set up your own custom menus to control filters, rather than using the built-in Filters menu. Finally, Strategy 4 sets up items that do not update until you, the document author, choose to update them.

Strategy 1: Creating R Outputs that do not use filters

The first strategy is to use a calculation in an R Output to generate the desired result. R calculations do not incorporate filters unless you deliberately build the filter into the calculation. R outputs will be unaffected when a view of your document applies a filter. Consider the following example of creating a table with R.

The table below on the left was created by dragging a variable from the Data tree onto a page. If a filter is applied in view mode this table will be automatically updated. The table on the right was created by inserting an R Output with R CODE of

cbind("%" = prop.table(table(Q3)) * 100)

and formatting the percent sign and decimals via the Appearance tab. It will not update when a viewer applies a filter, as the R CODE does not provide any instructions about what to do in the event of a filter. If the code was instead

cbind("%" = prop.table(table(Q3[QFilter])) * 100)

it would automatically be updated when a filter was applied.

Strategy 2: Removing QFilter from automatically-created R Outputs

Many analyses created in Displayr are based on R calculations. In edit mode, you can access their underlying code by clicking on the output and looking at Properties > R CODE in the Object Inspector. For example, the output below shows the R CODE of a regression model. When you look at the code, you will see that green formatting highlights where the code accesses any weights (QPopulationWeight) and filters (QFilter).

If you edit the code and replace QFilter with TRUE the code will ignore any filters. If you wish to add any filters to the code you can do so here as well; for example, replacing QFilter with gender == "Female" would cause the analyses to be based on data with a value of "Female" for the gender variable.

Strategy 3: Creating custom filters via controls

The automated filtering in Displayr view mode allows the user to apply filters by clicking on the Filters link at the top of the page. Alternatively, custom filter controls can be placed on a page and these can be selectively linked to different outputs. See How to Create an Interactive Infographic for a worked example.

Strategy 4: R Outputs with 'Automatic' unchecked

R Outputs only update if the Automatic checkbox at the top of the Object Inspector ticked (as it is in the example above). Thus, any R Output, including those created automatically (e.g., visualizations and machine learning models) can be prevented from being filtered by unchecking this. Of course, if you update your data then the output will not update to reflect this, so this strategy is dangerous. You would need to remember to manually update the outputs in your document when the data is updated.

This strategy can also be applied to tables created by dragging from the Data Tree, as follows:

• Create the table
• Hide the table (Appearance > Hide), so that it does not appear in your published document
• Look up the Name of the table by clicking in Properties > GENERAL in the Object Inspector
• Insert > R Output and set the R CODE to the name of the table (e.g., table.Q2) and press  Calculate 

This is effectively using R to make a copy of the table.

Settings

When you select the control, settings appear in the Control tab of the Object Inspector. Settings exist for controlling selected items, formatting, and tooltips.

In terms of functionality, the key settings are Item list and Selection mode. Item list should contain a semi-colon-delimited list of options from which the user can choose. For example: dog; cat; tiger. Selection mode governs whether a user can choose one or multiple items from the combo box.

The Properties tab of the Object Inspector has settings for the control's Name and LAYOUT settings. By default its name is Combo.box.

Using the combo box as an input into calculations

You can refer to combo boxes in R calculations. If you create an R Output (Insert > R Output (Analysis)) using the following code:

 
if(Combo.box == "dog") "A canine was selected" else "A feline was selected"

then this prints "A canine was selected" when selecting dog and "A feline was selected" otherwise. Prior to making a selection, the code returns Error: argument is of length zero. To deal with non-selection, we can use the following R code in the R Output:

 
if (length(Combo.box) == 0) "Nothing has been selected" else {
    if(Combo.box == "dog") "A canine was selected" else "A feline was selected"
}

Note that once something has been selected from a combo box where Selection mode is set to Single Selection, the user cannot de-select. So, if you were using the control to filter (e.g., Male; Female) and wanted to have a total option, you need to add it to the list (e.g., Male; Female; Total).

Whatever option(s) you selected when creating the document stay selected when a user accesses the document in View Mode. For example, if you select cat and then publish the dashboard, it will appear with cat selected.

Multiple selection

When the Selection mode is set to Multiple Selection, the user is presented with multiple check boxes. If you make this change to the example above, though, and select all three animals, you may be in for a bit of a surprise, with the result being "A canine was selected". This is because the way that the combo box control works is that any selections are returned as a vector. Thus, when nothing is selected, the length of the vector is 0. When selecting one item the length is 1. In this example, with three things selected, the length becomes 3. As the vector is greater than 0 (it's 3, in fact), then the first selected item in the list (which is "dog") will be interpreted as the one to use by the code, resulting in "A canine was selected".

Restrictions in View Mode

Some restrictions exist as to how combo boxes work in View Mode:

• When the user changes the combo box, any R Output that refers to this combo box will automatically update. However, any other R Outputs that refer to the R Output that refers to the combo box will not update. Thus, you need to put all your calculations in the R Output that refers to the combo box.
• You cannot refer to combo boxes in R Variables.
• R Outputs cannot refer to combo boxes on other pages.

To play around with this example, click here. My post titled How to Create an Online Choice Simulator contains a more ambitious example.

Click here for an interactive tutorial on interactive controls

Overfitting is one of the greatest challenges faced when training a model to predict outcomes.  In this blog post I show how to split data into a training and testing set, then how to fit support vector machines to the training set and assess their performance with the testing set. I use data about that delicious (for some), marine mollusc with brightly colored inner shells, the abalone. Our aim is to predict their gender from other physical characteristics.  One peculiar feature of abalone that concerns us here is that infants have undetermined gender, hence the outcome variable has 3 categories and not 2.

Splitting the data and training the model

First, we need to randomly split the data into a larger 70% training sample and a smaller 30% testing sample.  In Displayr we do this with Insert > Utilities > Filtering > Filters for Train-Test Split. This creates 2 new variables which can be used as filters. We are going to use the training sample to try different values of the support vector machine cost parameter and then check our models with the test sample.

In Displayr we add a support vector machine via Insert > More > Machine Learning > Support Vector Machine. I've chosen Length, Diameter, Whole weight and Rings as the Predictor variables, and Gender as the Outcome.  The other settings are left to their defaults as shown on the right. Finally to restrict the model to the training set, I change the filter (Home > Filter (Data Selection)) to Training sample. The result gives an overall accuracy of 56.77%, with a better performance for males and infants than females (as indicated by the bars).

Predicting with the test data

Although we now know the accuracy for predicting the genders in the training data, of more relevance is the accuracy on the 30% of the data that the support vector machine has not seen - the testing sample. To calculate this:

1. Select the output from the support vector machine.
2. Select Insert > More > Machine Learning > Diagnostic > Prediction-Accuracy Table.
3. With the prediction accuracy table selected, choose the Testing sample filter in Home > Filter (Data Selection).

The prediction-accuracy table illustrates the overlap between the categories which are predicted by the model and the original category of each case within the testing sample. The accuracy drops to 52.51%, as shown beneath the chart. The reduction is no surprise since the model has not seen this data before. Can we improve the performance of a support vector machine on unseen data?

Optimizing the cost

The cost parameter of a support vector machine determines how strictly it fits the training data. A high cost means the model attempts to make correct predictions on the training data even if that implies a complex relationship between the predictor and target variables. By contrast a low cost produces a simple model that is more prone to making mistakes on the training data. Higher cost leads to better accuracy on the training data, as shown in the table below. However this is not what we should be optimizing, rather we should aim to improve the accuracy on unseen data. Note that is it usual to explore a wide range of costs in a multiplicative manner, for example by increasing/decreasing in powers of 10. In Displayr, the cost parameter can be modified in the options for the support vector machine output (as shown in the screenshot above).

By increasing the cost we discover that test accuracy peaks at close to 55% with a cost of 1,000, then drops for higher values. This is overfitting in action. Predictions on the testing set are deteriorating at such high costs because the model is paying too much attention to aspects of the training data that are in fact random and do not help generalize to unseen examples. Looking at the test accuracy allows us to identify the point where the model has learned the essence of the relationship between the predictor and outcome variables but is not so focused that it is simply memorizing the training data. Another side-effect of high costs is that the model takes longer to run due to its complexity.

At the other end of the scale a cost of 0.001 leads to 37% testing accuracy and the Prediction-Accuracy Table shows that the model is so simple it predicts everything to be male, the most frequent category.  This is underfitting - the patterns of the data have not been learned well enough to make good predictions.

For the best performance we avoid both overfitting and underfitting, in this case with a cost of 1000.

TRY IT OUT
You can explore this data set for yourself in Displayr.

Photo credit: schoeband via VisualHunt.com / CC BY-NC-ND

Click here for an interactive tutorial on filtering

1. Creating simple filters from the ribbon

The simplest way to create a filter is to select Insert > Variables > Filter > New Filter (see below) and choose the variable or pair of variables that you wish to create a filter.

When you create a filter by pressing the buttons in the ribbon, you are both creating a filter and applying the filter to any currently selected outputs or pages. All the other methods described in this post create, but do not apply, filters.

2. Tagging a variable as a Filter

Any non-text variable set in Displayr can be set as a filter.

1. Select the variable in Data (bottom-left of the screen), and
2. Choose Usable as a filter from Properties (right-middle of the screen).

Cases that have a value of more than 0 are included in the filter.

3. Quickly creating lots of filters

To create lots of filters:

• Select the variable sets you wish to use to create the filters from Data (bottom-left of the screen).
• Insert > Variables > Filter > Filters from Selected Data.

A particularly useful aspect of the way that this works is that if you have merged together any of the categories in a variable set (see Introduction to Displayr 3: Creating tables, charts, and other visualizations), the new variables will be based on these merged categories. Also, if you wish you can modify these filters by changing their underlying JavaScript code, which we'll look at next.

4. Creating new R and JavaScript variables

New variables can be created using R and JavaScript (Insert > Variables). If these variables are tagged as filters (see method 2 above), they can be used as filters, with any values greater than 0 being included in the filter (i.e., when such variables are applied as filters, only the data for cases with values of more than 0 on the filter variable will be used).

A great example of how to create an R filter based off of the value in a drop down control box is on our blog here. For further examples of R-based filters, see How to Use Basic R Code for Creating Filters.

5. Variable Set Structure

This is a more exotic approach than the others, but can be a huge time saver. When the structure of a variable set is changed to either Binary - Multi or Binary - Grid the values of the variables are automatically recoded to be 0s and 1s, and you choose which values are converted to 1s and which to 0s. Then, select Usable as a filter for the variables.

Click here for an interactive tutorial on filtering

Example: Filtering Google NPS Data

Net Promoter Score (NPS) is a metric for evaluating the popularity of a brand.  The screenshot below shows a dashboard of Google’s NPS among men.  The cool thing about it is that this dashboard was created for the entire database, men and women combined.  Once it was finished, a single mouse click applied the Males filter to the whole page.  The charts and title updated and the decision tree was re-estimated. You can view the original dashboard in Displayr and collapse, expand and zoom in and out of the decision tree to see it better.

Here is the same dashboard again. This time it has been filtered to show the data for Females. As you can see, all the numbers and the decision tree have changed.

How to apply a filter in Displayr

1. Select whatever you want to filter. In the example above, I am selecting the entire page (from the list of pages at the top-left of the screen in Displayr.
2. In the Object Inspector on the right-hand side of the screen, select Inputs > FILTERS AND WEIGHT > Filter(s).
3. Choose the filter or filters that you want.

When choosing your filters, note that filters are grouped according to their variable set. Where two filters are in the same variable set, if they are both selected, it is treated as an OR operation. For example, if you were to choose Detractor and Passive from the list above, the data would be filtered to show people that are either Detractors or Passives. By contrast, where filters are in different variable sets, they are treated as AND operations. Thus, if you were to select Males and Females in the example above, the filtered data would contain nobody (as nobody is both male and female, in this data example).

You can examine this example in more detail by inspecting it in Displayr.

To learn more about all the different ways of creating filters, read the blog post 5 ways to create a filter in Displayr.