Variable Folders 🔗

In order to organize a large number of variables in a dataset, Crunch provides a hierarchical structure to group variables called “Folders”.

Basic concepts 🔗

Variables are in one and only one folder at all times.

There are a number of system folders that exist with every dataset created. These system folders cannot be renamed, moved, or deleted.

  • Root folder: The root folder is the top level folder where all other subfolders will be created by dataset editors. All the variables and subfolders here will be publicly accessible for all users with access to the dataset.
  • Hidden folder: This folder is a parallel top level folder separate from the root folder, which allows dataset editors to hide variables out of the public Root folder. “Hiddenness” is about display concerns, not authorization rules. API clients choose not to display hidden variables to users with view-only access, though they may use hidden variables in queries and may use filters and other assets derived from hidden variables.
  • Secure folder: Like the hidden folder, the secure folder is a special top-level folder. Secure variables are not shown in any form to users with view-only access to a dataset, and it is not possible for viewers to make queries with secure variables or access filters or other assets that reference secure variables.

Root 🔗

/datasets/{id}/folders/

Returns a Shoji catalog containing the root variable folder.

When a dataset is created without an order specified, the root folder will contain a flat list of all variables. As variables get grouped into folders, the root folder will continue showing any ungrouped variables in addition to the top-level folders.

Subfolder endpoint 🔗

/datasets/{id}/folders/{folder_id}/

All subfolder URLs are at a folder_id path segment below the root folder URL, regardless of how deeply nested inside other folders they may be. This means that a folder’s URL does not change when it is moved from one folder to another.

Folder payload 🔗

All folders types return a similar Shoji catalog with the following members:

Member Type Description
size int Indicates the number of total variables stored under this folder and its subfolders.
body object Contains a name attribute, which is the folder’s name. The root folder has an empty string as name.
index object Behaves like a standard Shoji catalog containing the children of the folder keyed by URL. This catalog will contain a mixed set of folder and variables URLs. Clients should differentiate each of the objects by looking at the tuple’s type attribute. Folders will have type “folder” while variables will contain their variables type. The size attribute for a folder type indicates the total number of variables stored under the folder and its subfolders.
graph list A flat list containing the list of URLs of variables and subfolders on the given folder.

JSON example:

{
    "element": "shoji:catalog",
    "self": "https://app.crunch.io/api/datasets/5ee0a0/folders/",
    "size": 10000,
    "body": {
        "name": "Folder name"
    },
    "index": {
        "https://app.crunch.io/datasets/abcdef/variables/123/": {
            "name": "Birth Year",
            "derived": false,
            "discarded": false,
            "secure": false,
            "hidden": false,
            "alias": "birthyear",
            "type": "numeric",
            "id": "123",
            "notes": "",
            "description": "In what year were you born?"
        },
        "https://app.crunch.io/datasets/abcdef/folders/qwe/": {
            "type": "folder",
            "name": "Subfolder name",
            "size": 99,
        },
        "https://app.crunch.io/datasets/abcdef/variables/456/": {
            "subvariables_catalog": "../variables/456/subvariables/",
            "name": "An Array",
            "derived": true,
            "discarded": false,
            "secure": false,
            "hidden": false,
            "alias": "arrayvar",
            "subvariables": [
                "../../variables/456/subvariables/439dcf/",
                "../../variables/456/subvariables/1c99ea/"
            ],
            "notes": "All variable types can have notes",
            "type": "categorical_array",
            "id": "456",
            "description": ""
        }
    },
    "graph": [
        "../variables/456/",
        "../variables/123/",
        "../folders/qwe/"
    ]
}

Creating folders 🔗

To create a variable folder, POST a folder representation to the URL of the folder in which you want the new one to be created (that is, the new folder will be a subfolder of it).

The payload must contain a body member containing only the name of the new subfolder. Note that all entities (variables and folders) collected in a folder must have unique names, but names do not need to be globally unique.

Additionally, an index member can be included containing URLs of the variables or folders that will be children of the new folder. The tuples associated with each variable should be an empty object. These variables or folders will be moved into the new folder from their previous locations.

An optional graph member indicates the order of entities within the folder. If included, it must contain all the items that the index contains.

{
   "entity": "shoji:catalog",
   "body": {
      "name": "New subfolder name"
   },
   "index": {
      "http://app.crunch.io/api/datasets/abc/variables/123/": {},
      "http://app.crunch.io/api/datasets/abc/variables/456/": {}
   },
   "graph": [
      "http://app.crunch.io/api/datasets/abc/variables/123/",
      "http://app.crunch.io/api/datasets/abc/variables/456/"
   ]
}

Moving folders and variables 🔗

In order to move folders or variables from one location to another, their URL must be included in the index catalog sent via a PATCH request to the destination folder.

The new elements will be moved out of their existing parents into the new folder location.

Inside a folder, subfolders and variables must be unique by name. Trying to move a folder or a variable that conflicts with the existing children of it will receive a 409 response from the server.

Reordering a folder’s contents 🔗

To reorder the elements inside a folder, PATCH the folder’s graph with a different order. The graph array must be a flat list and all the elements must be URLs that currently exist in the catalog’s index. New elements cannot be included just by adding them on the graph, though the PATCH request may also include an index to add items, as described above.

Deleting folders 🔗

Subfolders can be deleted by performing a DELETE request on their endpoints. It will effectively move the folder to the trash folder.

When deleting a folder, all the children will also be moved to the trash folder, including the variables that were part of it.

Moving a folder to the trash does not delete variables nor their data, but only makes them unavailable from the folder hierarchy.

Deleting a folder that is already in the trash does hard-delete that folder, including all variables & related data contained in that folder.

Trash 🔗

/datasets/{id}/folders/trash/

The trash folder is a special-purpose folder that lives outside the regular variable folders tree.

You can use the trash folder to “soft-delete” variables and folders.

Subfolders and variables inside the trash folder are only visible to dataset editors.

Performing a DELETE request to the trash endpoint will empty all items from the trash, hard-deleting the folders, variables, and related data.

Items in the trash folder may also be automatically hard-deleted after 24-48 hours.

Hidden 🔗

/datasets/{id}/folders/hidden/

The hidden folder is a special-purpose folder that lives outside the regular variable folders tree.

Variables and subfolders that should only be visible to dataset editors should be placed in the hidden folder.

To “hide” variables, PATCH the hidden folder’s index with the URLs of the variables. To “unhide” them, PATCH them to a different folder.

Secure 🔗

/datasets/{id}/folders/secure/

The secure folder is also a special-purpose folder that lives outside the regular variable folders tree.

To make variables secured from viewers’ access, PATCH the secure folder’s index with the URLs of the variables. A successful response returns 204 status.

Folders parents structure 🔗

/datasets/{id}/folders/parents/

Sometimes, clients need to build a folder hierarchy quickly given a specific folder URL. For that purpose we provide this endpoint so clients don’t have to discover a folder’s parents branch making multiple GET requests.

This read-only endpoint returns a shoji:catalog listing all the public folders that live under the root tree where each tuple has its name and the URL of its parent folder. The position attribute indicates the position on its parent’s graph.

{
    "element": "shoji:catalog",
    "self": "https://app.crunch.io/api/datasets/5ee0a0/folders/parents/",
    "index": {
        "../top1/": {
            "parent": "../",
            "name": "Top level folder",
            "position": 0
        },
        "../top2/": {
            "parent": "../",
            "name": "Top level folder as well",
            "position": 1
        },
        "../child1/": {
            "parent": "../top1/",
            "name": "Son of top1",
            "position": 2
        },
        "../gchild1/": {
            "parent": "../child1/",
            "name": "Son of child1",
            "position": 0
        },
        "../childa/": {
            "parent": "../top2/",
            "name": "Son of top2",
            "position": 3
        },
        "../child2/": {
            "parent": "../top1/",
            "name": "Son of top1",
            "position": 1
        }
    }
}

Filtering Folders by Variable Tags 🔗

The Root folder endpoint as well as the Subfolder endpoints can be filtered by Variable Tags, so that only “tagged” variables are exposed.

In contrast with the normal behavior, empty Folders are not exposed when Variable Tags are applied.

In order to specify the tags to filter by, one or more tag=<tag_name> querystring parameters must be added to the Folder GET requests.

The following are examples of tag filtering:

  • /datasets/{id}/folders/?tag=tag1
  • /datasets/{id}/folders/?tag=tag1&tag=tag%202
  • /datasets/{id}/folders/{subfolder_id}/?tag=tag1
  • /datasets/{id}/folders/{subfolder_id}/?tag=tag1&tag=tag2

When filtering by multiple tags, the set of tagged variables will be constructed using AND logic. That is, when filtering by tag=tag1&tag=tag2, only variables that are tagged both tag1 and tag2 will be exposed.

The structure of the filtered Folder endpoints is exactly the same as their unfiltered counterparts.

If one or more of the tag=<tag_name> parameters are invalid, the Folder endpoint returns a 400 Bad Request response with a helpful message on the reason for the failure to apply the tag(s).