Adding a subsequent Source 🔗

Once you have created a Dataset, you can upload new files and append rows to the same Dataset as often as you like. If the structure of each file is the same as that of the first uploaded file, Crunch should automatically pass your new rows through exactly the same process as the old rows. If there are any derived variables in your Dataset, new data will be derived in the new rows following the same rules as the old data. You can follow the progress as above via the batch’s status attribute.

Let’s look at an example: you had uploaded an initial CSV of 3 columns, A, B and C. Then:

• The Crunch system automatically converted column A from the few strings that were found in it to a Categorical type.
• You derived a new column D that consisted of B * C.

Then you decide to upload another CSV of new rows. What will happen?

When you POST to create the second Batch, the service will: 1) match up the new A with the old A and cast the new strings to existing categories by name, and 2) fill column D for you with B * C for each new row.

However, from time to time, the new source has significant differences: a new variable, a renamed variable, and other changes. When you append the first Source to a Dataset, there is nothing with which to conflict. But a subsequent POST to batches/ may result in a conflict if the new source cannot be confidently reconciled with the existing data. Even though you get a 201 Created response for the new batch resource, it will have a status of “conflict”.

Reporting and Resolving Conflicts 🔗

When you append a Source to an existing Dataset, the system attempts to match up the new data with the old. If the source’s schema can be automatically aligned with the target Dataset, the new rows from the Batch are appended. When things go wrong, however, the Batch can be inspected to see what conflicted with the target (or vice-versa, in some cases!).

GET the new Batch:

GET /api/datasets/{dataset_id}/batches/{batch_id}/ HTTP/1.1
...
--------
200 OK
Content-Type: application/shoji

{
    "element": "shoji:entity",
    "body": {
        "conflicts": {
          "cdbd11/": {
            "metadata": {},
            "conflicts": [{
              "message": "Types do not match and cannot be converted",
            }]
          }
        }
    }
}

If any variable conflicts, it will possess one or more “conflicts” members. For example, if the new variable “cdbd11” had a different type that could not be converted compared to the existing variable “cdbd11”, the Batch resource would contain the above message. Only unresolvable conflicts will be shown; if a variable is not reported in the conflicts object, it appended cleanly.

See Batches for more details on batch entities and conflicts.

Streaming rows 🔗

Existing datasets are best sent to Crunch as a single Source, or a handful of subsequent Sources if gathered monthly or on some other schedule. Sometimes however you want to “stream” data to Crunch as it is being gathered, even one row at a time, rather than in a single post-processing phase. You do not want to make each row its own batch (it’s simply not worth the overhead). Instead, you should make a Stream and send rows to it, then periodically create a Source and Batch from it.

{"var_id_1": 1, "var_id_2": "a"}

by_alias = ds.variables.by('alias')
while True:
    row = my_system.read_a_row()
    importing.importer.stream_rows(ds, {
        'gender': row['gender'],
        'age': row['age']
    })

{
    "stream": null,
    "type": "ldjson",
    "name": "My streamed rows",
    "description": "Yet Another batch from the stream"
}

ds.batches.create({"body": {
    "stream": None,
    "type": "ldjson",
    "name": "My streamed rows",
    "description": "Yet Another batch from the stream"
}})

Transformations during combination 🔗

The combine procedures will perform normal append matching rules which means that any mismatch on aliases or types will not proceed, as well limiting the existing union of variables from the present datasets as the result.

It is possible to provide transformations on the datasets to ensure that they line up on the combination phase and to add extra columns with constant dataset metadata per dataset on the resulting combined result.

Each {"dataset"} argument allows for an extra frame key that can contain a function expression on the desired dataset transformation, for example:

{
    "dataset": "<dataset_url>",
    "frame": {
        "function": "select",
        "args": [{
            "map": {
                "*": {"variable": "*"},
                "dataset_id": {
                    "value": "<dataset_id>",
                    "type": "text",
                    "references": {
                        "name": "Dataset ID",
                        "alias": "dataset_id"
                    }
                }
            }
        }]
    }
}

Selecting a subset of variables to combine 🔗

In the same fashion that it is possible to add extra variables to the dataset transforms, it is possible to select which variables only to include.

Note in the example above, we use the "*": {"variable": "*"} expressions which instructs the server to include all variables. Omitting that would cause to only include the selected variables, for example:

{
    "dataset": "<dataset_url>",
    "frame": {
        "function": "select",
        "args": [{
            "map": {
                "A": {"variable": "A"},
                "B": {"variable": "B"},
                "C": {"variable": "C"},
                "dataset_id": {
                    "value": "<dataset_id>",
                    "type": "text",
                    "references": {
                        "name": "Dataset ID",
                        "alias": "dataset_id"
                    }
                }
            }
        }]
    }
}

On this example, the expression indicates to only include variables with IDs A, B and C from the referenced dataset as well as add the new extra variable dataset_id. This would effectively append only these 4 variables instead of the full dataset’s variables.