Variables 🔗

GET catalog 🔗

When authenticated and authorized to view the given dataset, GET returns 200 status with a Shoji Catalog of variables in the dataset. If authorization is lacking, response will instead be 404.

Array subvariables are not included in the index of this catalog. Their metadata are instead accessible in each array variable’s “subvariables_catalog”.

Private variables are not included in the index of this catalog, although entities may be present at variables/{id}/. See Private Variables for an index of those.

Catalog tuples contain the following keys:

The catalog has two optional query parameters:

With the relative flag enabled, the variable catalog looks something like this:

{
    "element": "shoji:catalog",
    "self": "https://app.crunch.io/api/datasets/5ee0a0/variables/",
    "orders": {
        "hier": "https://app.crunch.io/api/datasets/5330a0/variables/hier/",
        "personal": "https://app.crunch.io/api/datasets/5330a0/variables/personal/",
        "weights": "https://app.crunch.io/api/datasets/5ee0a0/variables/weights/"
    },
    "specification": "https://app.crunch.io/api/specifications/variables/",
    "description": "List of Variables of this dataset",
    "index": {
        "a77d9f/": {
            "name": "Birth Year",
            "derived": false,
            "discarded": false,
            "alias": "birthyear",
            "type": "numeric",
            "id": "a77d9f",
            "notes": "",
            "description": "In what year were you born?"
        },
        "9e4c84/": {
            "name": "Comments",
            "derived": false,
            "discarded": false,
            "alias": "qccomments",
            "type": "text",
            "id": "9e4c84",
            "notes": "Global notes about this variable.",
            "description": "Do you have any comments on your experience of taking this survey (optional)?"
        },
        "aad4ad/": {
            "subvariables_catalog": "aad4ad/subvariables/",
            "name": "An Array",
            "derived": true,
            "discarded": false,
            "alias": "arrayvar",
            "subvariables": [
                "439dcf/",
                "1c99ea/"
            ],
            "notes": "All variable types can have notes",
            "type": "categorical_array",
            "id": "aad4ad",
            "description": ""
        }
    }
}

PATCH catalog 🔗

Use PATCH to edit the “name”, “description”, “alias”, or “discarded” state of one or more variables. A successful request returns a 204 response. The attributes changed will be seen by all users with access to this dataset; i.e., names, descriptions, aliases, and discarded state are not merely attributes of your view of the data but of the datasets themselves.

Authorization is required: you must have “edit” privileges on the dataset being modified, as shown in the “permissions” object in the dataset’s catalog tuple. If you try to PATCH and are not authorized, you will receive a 403 response and no changes will be made.

The tuple attributes other than “name”, “description”, “alias”, and “discarded” cannot be modified here by PATCH. Attempting to modify other attributes, or including new attributes, will return a 400 response. Variable “type” can only be modified by the “cast” method, described below. The “subvariables” can be modified by PATCH on the variable entity. “subvariables_catalog” is a URL to a different variable catalog and is thus not editable, though you can navigate to its location and modify subvariable attributes there. A variable’s “id” and its “derived” state are immutable.

When PATCHing, you may include only the keys in each tuple that are being modified, or you may send the complete tuple. As long as the keys that cannot be modified via PATCH here are not modified, the request will succeed.

Note that, because this catalog contains its entities (rather than collecting them), you cannot PATCH to add new variables, nor can you PATCH a null tuple to delete them. Attempting either will return a 400 response. Creating variables is allowed only by POST to the catalog, while deleting variables is accomplished via a DELETE on the variable entity.

{
    "element": "shoji:catalog",
    "index": {
        "9e4c84/": {
            "discarded": true
        }
    }
}

PATCHing this payload on the above catalog will return a 204 status. A subsequent GET of the catalog returns the following response; note the change in line 24.

{
    "element": "shoji:catalog",
    "self": "https://app.crunch.io/api/datasets/5ee0a0/variables/",
    "orders": {
        "hier": "https://app.crunch.io/api/datasets/5330a0/variables/hier/",
        "personal": "https://app.crunch.io/api/datasets/5330a0/variables/personal/",
        "weights": "https://app.crunch.io/api/datasets/5ee0a0/variables/weights/"
    },
    "specification": "https://app.crunch.io/api/specifications/variables/",
    "description": "List of Variables of this dataset",
    "index": {
        "a77d9f/": {
            "name": "Birth Year",
            "derived": false,
            "discarded": false,
            "alias": "birthyear",
            "type": "numeric",
            "id": "a77d9f",
            "notes": "",
            "description": "In what year were you born?"
        },
        "9e4c84/": {
            "name": "Comments",
            "derived": false,
            "discarded": true,
            "alias": "qccomments",
            "type": "text",
            "id": "9e4c84",
            "notes": "Global notes about this variable.",
            "description": "Do you have any comments on your experience of taking this survey (optional)?"
        },
        "aad4ad/": {
            "subvariables_catalog": "aad4ad/subvariables/",
            "name": "An Array",
            "derived": true,
            "discarded": false,
            "alias": "arrayvar",
            "subvariables": [
                "439dcf/",
                "1c99ea/"
            ],
            "notes": "All variable types can have notes",
            "type": "categorical_array",
            "id": "aad4ad",
            "description": ""
        }
    }
}

POST catalog 🔗

A POST to this resource must be a Shoji Entity with the following “body” attributes:

• name
• type
• If “type” is “categorical”, “multiple_response”, or “categorical_array”: categories: an array of category definitions
• If “type” is “multiple_response” or “categorical_array”: subvariables: an array of URLs of variables to be “bound” together to form the array variable
• If “type” is “multiple_response” or “categorical_array”: subreferences: an object keyed by each of the subvariable URLs where each value contains partial variable definitions, which will be created as categorical subvariables of the array. If included, the array definition must include “categories”, which are shared among the subvariables.
• If type is “multiple_response”, the definition may include selected_categories: an array of category names present in the subvariables. This will mark the specified category or categories as the “selected” response in the multiple response variable. If no “selected_categories” array is provided, the new variable will use any categories already flagged as “selected”: true. If no such category exists, the response will return a 400 status.
• If “type” is “datetime”: resolution: a string, such as “Y”, “Q”, “M”, “W”, “D”, “h”, “m”, “s”, “ms”, that indicates the unit size of the datetime data.
• folder: optional, a URL of a variable folder in the dataset into which the new variable should be placed. If omitted, the variable will be created on the root folder. Personal variables (with "private": true) cannot be placed in a folder; attempting to do so returns 400 status.

See Variable Definitions for more details and examples of valid attributes, and Feature Guide: Array Variables for more information on the various cases for creating array variables.

It is encouraged, but not required, to include an “alias” in the body. If omitted, one will be generated from the required “name”.

You may also include “values”, which will create the column of data corresponding to this variable definition. See Importing Data: Column-by-column for details and examples.

You may instead also include a “derivation” to derive a variable as a function of other variables. In this case, “type” is not required because it depends on the output of the specified derivation function. For details and examples, see Deriving Variables.

A 201 indicates success and includes the URL of the newly-created variable in the Location header.

Private variables catalog 🔗

/datasets/{id}/variables/private/{?relative}

GET returns a Shoji Catalog of variables, as described above, containing those variables that are private to the authenticated user. You may PATCH this catalog to edit names, aliases, descriptions, etc. of the private variables. POST, however, is not supported at this endpoint. To create new personal variables, POST to the main variables catalog with a "private": true body attribute.

PATCH to this catalog allows dataset editors to make variables private by sending a Shoji catalog payload containing the URLs of the variables they wish to make personal.

The Shoji catalog should have empty objects as tuples.

{
    "element": "shoji:catalog",
    "index": {
      "http://app.crunch.io/api/datasets/abc/variables/xyz/": {}
    }
}

GET 🔗

Returns a Shoji Order.

PATCH 🔗

Will expect a Shoji Order representation containing a replacement or new grouped entities. This allows one to create new groups on the fly or overwrite existing groups with new ‘entities’.

The match happens by each group name and will overwrite the values of each group with the received one.

After PATCH any variable not present in the order will always be appended to the root of the graph.

PUT 🔗

Receives a Shoji Order representation with a completely new graph. Any previously existing group will be eliminated and any new groups will be added. This will overwrite the complete set of current groups.

After PUT any variable not present on any of the groups will always be appended to the root of the graph.

GET 🔗

Returns a Shoji Order for this user.

PATCH 🔗

Same as hierarchical order, receives a Shoji Order representation to overwrite the existing order. Personal variables are allowed here.

PUT 🔗

Behaves sames as PATCH.

GET 🔗

GET a shoji:order that contains the urls of the variables that have been designated as possible weight variables.

PATCH 🔗

PATCH the graph with a list of the desired list of weight variables. The list will always be overwritten with the new values. This order can only be a flat list of URLs, any nesting will be rejected with a 400 response.

If the dataset has a default weight variable configured, it will always be present on the response even if it wasn’t included on a PATCH request.

Removing variables from this list will have the side effect of changing any user’s preference that had such variables set as their weight to the current dataset’s default weight.

Only numeric variables are allowed to be used as weight. If a variable of another type is included in the list, the server will abort and return a 409 response.

{
  "graph": ["https://app.crunch.io/api/datasets/42d0a3/variables/42229f"]
}

PUT 🔗

Behaves sames as PATCH.

GET 🔗

Variable entities’ body attributes contain the following:

Category objects have the following members:

Format objects may contain:

View objects may contain:

Variable entities’ catalog attributes contain the folder member that points to the variable’s containing folder.

Additionally, the variable entity will hold references to related resources

PATCH 🔗

PATCH variable entities to edit their metadata. Send a Shoji Entity with a “body” member containing the attributes to modify. Omitted body attributes will be unchanged.

Successful requests return 204 status. Among the actions achievable by PATCHing variable entities:

• Editing category attributes and adding categories. Include all categories.
• Remove categories by sending all categories except for the ones you wish to remove. You can only remove categories that don’t have any corresponding data values. Attempting to remove categories that have data associated will fail with a 400 response status.
• Reordering or removing subvariables in an array. Unlike categories, subvariables cannot be added via PATCH here.
• Editing derivation expressions
• Editing format and view settings
• Changing a datetime variable’s resolution

Actions that are best or only achieved elsewhere include:

• changing variable names, aliases, and descriptions, which is best accomplished by PATCHing the variable catalog, as described above;
• changing a variable’s type, which can only be done by POSTing to the variable’s “cast” resource (see Convert type below);
• editing names, aliases, and descriptions of subvariables in an array, which is done by PATCHing the array’s subvariable catalog;
• altering missing rules.

Variable “id” and “dataset_id” are immutable.

Example:

{
  "subvariables": [
    "http://app.crunch.io/api/datasets/d4db9831e08a4922b054e49b47a0045c/variables/00000c/subvariables/0008/",
    "http://app.crunch.io/api/datasets/d4db9831e08a4922b054e49b47a0045c/variables/00000c/subvariables/0007/",
    "http://app.crunch.io/api/datasets/d4db9831e08a4922b054e49b47a0045c/variables/00000c/subvariables/0009/"
  ],
  "subreferences": {
    "http://app.crunch.io/api/datasets/d4db9831e08a4922b054e49b47a0045c/variables/00000c/subvariables/0008/": {
      "alias": "subvar_2",
      "name": "v2_new_name",
      "description": null
    },
    "http://app.crunch.io/api/datasets/d4db9831e08a4922b054e49b47a0045c/variables/00000c/subvariables/0007/": {
      "alias": "subvar_1_new_name",
      "name": "v1_new_name",
      "description": null
    },
    "http://app.crunch.io/api/datasets/d4db9831e08a4922b054e49b47a0045c/variables/00000c/subvariables/0009/": {
      "alias": "subvar_3",
      "name": "subvar_3",
      "description": "new description"
    }
  }
}

POST 🔗

Calling POST on an array resource will “unbind” the variable. On success, POST returns 200 status with a Shoji View, containing the URLs of the (formerly sub-)variables, which are promoted to regular variables.

Trying to unbind a variable that is not an array will return a 400 response from the server.

A derived array cannot be unbound. It must first be integrated (by PATCHing null to its derivation expression, making it non-derived for good) and may then be unbound. Since this “undoes” the array, you should first see if there’s a way to refer to either a subvariable of the derived array, or one of the variables or subvariables from which it is derived, rather than unbinding.

DELETE 🔗

Calling DELETE on this resource will delete the variable. On success, DELETE returns 200 status with an empty Shoji View. Deleting an array deletes all its subvariable data as well.

numeric 🔗

The members include several counts:

• count: The number of entries in the variable.
• valid_count: The number of entries in the variable which are not missing.
• missing_count: The number of entries in the variable which are missing.
• missing_frequencies: An array of row objects. Each row represents a distinct missing reason, and includes the reason phrase as the “value” member and the number of entries which are missing for that reason as the “count” member.
• histogram: An array of row objects. Each row represents a discrete interval in the probability distribution, whose boundaries are given by the “bins” pair. An “at” member is included giving the midpoint between the two boundaries. The “value” member gives a count of entries which fall into the given bin. as well as basic summary statistics:
• fivenum: An array of five [quartile, point] pairs, where the “quartile” element is one of the strings “0”, “0.25”, “0.5”, “0.75”, “1”, representing the min, first quartile, median, third quartile, and max boundaries to divide the data values into four equal groups. The “point” is the real number at each boundary, and is estimated using the same algorithm as Excel or R’s “algorithm 7”, where h is: (N - 1)p + 1.
• min, median, max: taken from “fivenum”, above.
• mean: the sum of the values divided by the number of values, or, if weighted, the sum of weight times value divided by the sum of the weights.
• stddev: The standard deviation of the values.

categorical 🔗

The basic counts are included:

• count: The number of entries in the variable.
• valid_count: The number of entries in the variable which are not missing.
• missing_count: The number of entries in the variable which are missing.
• missing_frequencies: An array of row objects. Each row represents a distinct missing reason, and includes the reason phrase as the “value” member. The number of entries which are missing for that reason is included as the “count” member.

And the typical “frequencies” member is expanded into a custom “categories” member:

• categories: An array of row objects. Each row represents a distinct category (whether valid or missing), and includes its id the _id member (note the leading underscore), and its name as the “name” member. The “missing” member is true or false depending on whether the category is marked missing or not. The number of entries which possess that value is included as the “count” member.

text 🔗

The basic counts are included:

• count: The number of entries in the variable.
• valid_count: The number of entries in the variable which are not missing.
• missing_count: The number of entries in the variable which are missing.
• nunique: The number of distinct values in the data.
• sample: A sample of 5 entries of the data.

In addition:

• max_chars: The number of characters of the longest value in the data.

Univariate frequencies 🔗

/datasets/{id}/variables/{id}/frequencies/{?filter,exclude_exclusion_filter}

An array of row objects, giving the count of distinct values. The exact members vary by type:

• numeric: Each row represents a distinct valid value, and includes it as the “value” member. The number of entries which possess that value is included as the “count” member.
• categorical: Each row represents a distinct category (whether valid or missing), and includes its id the _id member (note the leading underscore), and its name as the “name” member. The “missing” member is true or false depending on whether the category is marked missing or not. The number of entries which possess that value is included as the “count” member.
• text: Each row represents a distinct valid value, and includes it as the “value” member. The number of entries which possess that value is included as the “count” member. The length of the array is limited to 10 entries; if more than 10 distinct values are present in the data, an 11th row is added with a “value” member of “(Others)”, summing their counts.

Convert type 🔗

/datasets/{id}/variables/{id}/cast/

A POST to this resource, with a JSON request body of {“cast_as”: type}, will alter the variable to the given type. If the variable cannot be cast to the given type, 409 is returned. See next to obtain a preview summary of such a cast before committing to it.

Missing values 🔗

/datasets/{id}/variables/{id}/missing_rules/

A Shoji Entity whose “body” member contains an array of missing rule objects. POST a {reason: rule} to this URL to add a new rule. Rules take one of the following forms:

• {‘value’: v}: Entries which match the given value will be marked as missing for the given reason.
• {‘set’: [v1, v2, …]}: Entries which are present in the given set will be marked as missing for the given reason.
• {‘range’: [lower, upper], ‘inclusive’: [true, false]}: Entries which exist between the given boundaries will be marked as missing for the given reason. If either “inclusive” element is null, the corresponding boundary is unbounded.
• {‘function’: ‘…’, ‘args’: […]}: Entries which match the given filter function will be marked as missing for the given reason. This is typically a tree of simple rules logical-OR’d together.

Example:

[
  {
    "Invalid": {"value": 0},
    "Sarai doesn't know how to use a calculator :(": {"range": [1000, null], "inclusive": [true, false]}
  }
]

Subvariables 🔗

/datasets/{id}/variables/{id}/subvariables/

Values 🔗

/datasets/{id}/variables/{id}/values/{?start,total,filter}

A GET on this set of resources will return a JSON array of values from the variable’s data. Numeric variables will return numbers, text variables will return strings, and categorical variables will return category names for valid categories and {“?”: code} missing markers for missing categories. The “start” and “total” parameters paginate the results. The “filter” is a Crunch filter expression.

Note that this endpoint is only accessible by dataset editors unless the viewers_can_export dataset setting is set to true, else the server will return a 403 response.