Variable Folders 🔗

In order to organize a large number of variables in a dataset, Crunch provides a hierarchical structure to group variables called “Folders”.

Basic concepts 🔗

There are a number of system folders that exist with every dataset created. These system folders cannot be renamed, moved, or deleted.

  • Root folder: The root folder is the top level folder where all other subfolders will be created by dataset editors. All the variables and subfolders here will be publicly accessible for all users with access to the dataset.
  • Hidden variables folder: This folder is a parallel top level folder separate from the root folder, which allows dataset editors to hide variables out of the public Root folder. This tree structure is only accessible for dataset editors.

Variables can only be in one folder. Variables can be in any of the top level system folders or in any subfolder of them at any given time, but they can never be in two folders at the same time.

Root 🔗

/datasets/{id}/folders/

Returns a Shoji catalog containing the root variable folder.

When a dataset is created without an order specified, the root folder will contain a flat list of all variables. As variables get grouped into folders, the root folder will continue showing any ungrouped variables in addition to the top-level folders.

Subfolder endpoint 🔗

/datasets/{id}/folders/{folder_id}/

All subfolder URLs are at a folder_id path segment below the root folder URL, regardless of how deeply nested inside other folders they may be. This means that a folder’s URL does not change when it is moved from one folder to another.

Folder payload 🔗

All folders types return a similar Shoji catalog with the following members:

Member Type Description
size int Indicates the number of total variables stored under this folder and its subfolders.
body object Contains a name attribute, which is the folder’s name. The root folder has an empty string as name.
index object Behaves like a standard Shoji catalog containing the children of the folder keyed by URL. This catalog will contain a mixed set of folder and variables URLs. Clients should differentiate each of the objects by looking at the tuple’s type attribute. Folders will have type “folder” while variables will contain their variables type. The size attribute for a folder type indicates the total number of variables stored under the folder and its subfolders.
graph list A flat list containing the list of URLs of variables and subfolders on the given folder.

JSON example:

{
    "element": "shoji:catalog",
    "self": "https://app.crunch.io/api/datasets/5ee0a0/folders/",
    "size": 10000,
    "body": {
        "name": "Folder name"
    },
    "index": {
        "https://app.crunch.io/datasets/abcdef/variables/123/": {
            "name": "Birth Year",
            "derived": false,
            "discarded": false,
            "alias": "birthyear",
            "type": "numeric",
            "id": "123",
            "notes": "",
            "description": "In what year were you born?"
        },
        "https://app.crunch.io/datasets/abcdef/folders/qwe/": {
            "type": "folder",
            "name": "Subfolder name",
            "size": 99,
        },
        "https://app.crunch.io/datasets/abcdef/variables/456/": {
            "subvariables_catalog": "../variables/456/subvariables/",
            "name": "An Array",
            "derived": true,
            "discarded": false,
            "alias": "arrayvar",
            "subvariables": [
                "../../variables/456/subvariables/439dcf/",
                "../../variables/456/subvariables/1c99ea/"
            ],
            "notes": "All variable types can have notes",
            "type": "categorical_array",
            "id": "456",
            "description": ""
        }
    },
    "graph": [
        "../variables/456/",
        "../variables/123/",
        "../folders/qwe/"
    ]
}

Creating folders 🔗

To create a variable folder, POST a folder representation to the URL of the folder in which you want the new one to be created (that is, the new folder will be a subfolder of it).

The payload must contain a body member containing only the name of the new subfolder. Note that all entities (variables and folders) collected in a folder must have unique names, but names do not need to be globally unique.

Additionally, an index member can be included containing URLs of the variables or folders that will be children of the new folder. The tuples associated with each variable should be an empty object. These variables or folders will be moved into the new folder from their previous locations.

An optional graph member indicates the order of entities within the folder. If included, it must contain all the items that the index contains.

{
   "entity": "shoji:catalog",
   "body": {
      "name": "New subfolder name"
   },
   "index": {
      "http://app.crunch.io/api/datasets/abc/variables/123/": {},
      "http://app.crunch.io/api/datasets/abc/variables/456/": {}
   },
   "graph": [
      "http://app.crunch.io/api/datasets/abc/variables/123/",
      "http://app.crunch.io/api/datasets/abc/variables/456/"
   ]
}

Moving folders and variables 🔗

In order to move folders or variables from one location to another, their URL must be included in the index catalog sent via a PATCH request to the destination folder.

The new elements will be moved out of their existing parents into the new folder location.

Inside a folder, subfolders and variables must be unique by name. Trying to move a folder or a variable that conflicts with the existing children of it will receive a 409 response from the server.

Reordering a folder’s contents 🔗

To reorder the elements inside a folder, PATCH the folder’s graph with a different order. The graph array must be a flat list and all the elements must be URLs that currently exist in the catalog’s index. New elements cannot be included just by adding them on the graph, though the PATCH request may also include an index to add items, as described above.

Deleting folders 🔗

Subfolders can be deleted by performing a DELETE request on their endpoints. It will effectively move the folder to the trash folder.

When deleting a folder, all the children will also be moved to the trash folder, including the variables that were part of it.

Moving a folder to the trash does not delete variables nor their data, but only makes them unavailable from the folder hierarchy.

Deleting a folder that is already in the trash does hard-delete that folder, including all variables & related data contained in that folder.

Trash 🔗

/datasets/{id}/folders/trash/

The trash folder is a special-purpose folder that lives outside the regular variable folders tree.

You can use the trash folder to “soft-delete” variables and folders.

Subfolders and variables inside the trash folder are only visible to dataset editors.

Performing a DELETE request to the trash endpoint will empty all items from the trash, hard-deleting the folders, variables, and related data.

Items in the trash folder may also be automatically hard-deleted after 24-48 hours.

Hidden 🔗

/datasets/{id}/folders/hidden/

The hidden folder is a special-purpose folder that lives outside the regular variable folders tree.

Variables and subfolders that should only be visible to dataset editors should be placed in the hidden folder.

Folders parents structure 🔗

/datasets/{id}/folders/parents/

Sometimes, clients need to build a folder hierarchy quickly given a specific folder URL. For that purpose we provide this endpoint so clients don’t have to discover a folder’s parents branch making multiple GET requests.

This read-only endpoint returns a shoji:catalog listing all the public folders that live under the root tree where each tuple has its name and the URL of its parent folder. The position attribute indicates the position on its parent’s graph.

{
    "element": "shoji:catalog",
    "self": "https://app.crunch.io/api/datasets/5ee0a0/folders/parents/",
    "index": {
        "../top1/": {
            "parent": "../",
            "name": "Top level folder",
            "position": 0,
        },
        "../top2/": {
            "parent": "../",
            "name": "Top level folder as well",
            "position": 1,
        },
        "../child1/": {
            "parent": "../top1/",
            "name": "Son of top1",
            "position": 2,
        },
        "../gchild1/": {
            "parent": "../child1/",
            "name": "Son of child1",
            "position": 0,
        },
        "../childa/": {
            "parent": "../top2/",
            "name": "Son of top2",
            "position": 3,
        },
        "../child2/": {
            "parent": "../top1/",
            "name": "Son of top1",
            "position": 1,
        },
}