create_dataset and add_file error out

[Danny-dK]

I'm new to the dataverse package. I can't seem to get things working and I don't understand any of the errors.

The following functions are working:

get_dataverse('mydatversename')
contents <- dataverse_contents('mydataversename')
get_dataset(contents[[1]])

But adding datafiles to a newly created dataset fails (i'm following the examples given here https://www.rdocumentation.org/packages/dataverse/versions/0.3.0):

library(dataverse)
Sys.setenv("DATAVERSE_SERVER" = "demo.dataverse.nl")
Sys.setenv("DATAVERSE_KEY" = "myapitokenfromdataverse")

# retrieve your service document
d <- service_document()

# create a list of metadata
metadat <-
  list(
    title       = "My Study3",
    creator     = "Doe, John",
    description = "An example study"
  )

# create the dataset
ds <- initiate_sword_dataset("mydataversename", body = metadat)

# add files to dataset
tmp <- tempfile()
write.csv(iris , file = tmp)
f <- add_file(ds, file = tmp)

The add_file() fails with the error:

Error in handle_url(handle, url, ...) : 
  Must specify at least one of url or handle

The following also does not work

f <- add_dataset_file(file = tmp, dataset = ds)
Error in parse_url(url) : length(url) == 1 is not TRUE

The other example at the previously mentioned link using the native API throws an error at the very first line and thus can't even create the dataset:

metadat <-
  list(
    title       = "My Study4",
    creator     = "Doe, John",
    description = "An example study"
  )

ds <- create_dataset('mydataversename', body = metadat)

Error in create_dataset("mydataversename", body = metadat) : 
  Internal Server Error (HTTP 500).

Can anyone tell me why it is not working to add files? I'm not particularly adept at error handling, so if you need more info please instruct me on how to receive that info.

R version 4.03
Rstudio 1.4.1103
Windows 1909 build 18363.1256
demo.dataverse.nl v. 4.18.1
dataverse package v 0.3.0

[Danny-dK]

I am able to get it working by converting the curl command (https://guides.dataverse.org/en/5.3/api/native-api.html#create-dataset-command) to R code using this online tool to convert curl to R https://curl.trillworks.com/#r

Curl as displayed in https://guides.dataverse.org/en/5.3/api/native-api.html#create-dataset-command at the "Add a File to a Dataset" section:

curl -H X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST -F file=@data.tsv -F 'jsonData={"description":"My description.", "restrict":"false"}' "https://demo.dataverse.nl/api/datasets/:persistentId/add?persistentId=doi:[mydoicreated]&version=DRAFT"

becomes R:

require(httr)

headers = c(
  `X-Dataverse-key` = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
)

params = list(
  `persistentId` = 'doi:[mydoicreated]',
  `version` = 'DRAFT'
)

files = list(
  `file` = upload_file('data.tsv'),
  `jsonData` = '{"description":"My description.", "restrict":"false"}'
)

res <- httr::POST(url = 'https://demo.dataverse.nl/api/datasets/:persistentId/add', httr::add_headers(.headers=headers), query = params, body = files)

The R code now works using httr library. Why would it not work with the dataverse package?

EDIT::
F me. The HTTR code got me thinking on the reference to the dataset name. Originally I had:

# create the dataset
ds <- initiate_sword_dataset("mydataversename", body = metadat)

# add files to dataset
tmp <- tempfile()
write.csv(iris , file = tmp)
f <- add_file(ds, file = tmp)

The add_file did not work nor did add_dataset_file. However changing it do include the full doi does work:

f <- add_dataset_file(file = tmp, dataset = 'doi:[mydoicreated]&version=DRAFT')

but this only works for add_dataset_file, not for add_file. The create_dataset() function still produces the same error. So for now I can create a dataset with the 'inititate_sword_dataset()' function and add files to it with add_dataset_file() as long as I give the full doi of that dataset (referencing it with just the dataset name created, doesn't seem to work).

[Danny-dK]

As per request.
To summarize, my code that now works is:

# Load required libraries.

library(dataverse)
library(rstudioapi)
library(dplyr)


# Set variables
 
Sys.setenv("DATAVERSE_SERVER" = "demo.dataverse.nl")
Sys.setenv("DATAVERSE_KEY" = askForPassword())


# Set working directory to where you opened this saved R file. (It is essentially somewhat similar to what the 'here' package does)

setwd(dirname(getActiveDocumentContext()$path))


# List local files to upload (from a folder called dataversetest on my hard-drive)
 
files <-  list.files(path = 'dataversetest/', full.names = TRUE)


# Retrieve service document

d <- service_document()


# Specify the metadatafields as objects.

metatitle <- paste0("thisistheRtestmult_", format(Sys.time(), '%Y%m%d_%H%M'))
metacreator <- "Doe, John"
metadescription <- "AnRtestupload"


# Create the actual metadata list.

metadat <- list(
              title       = metatitle,
              creator     = metacreator,
              description = metadescription)


# Create a dataset. The dataset created will get the name as specified in the 'title' entry of the metadata listed before. 

ds <- initiate_sword_dataset(dataverse = "UU", body = metadat)


# Search for the doi of the just created dataset. This is required to upload files using add_dataset_file in the next line after. 
# Specifying the ds object created in the line above as the dataset in the add_dataset_file function does not work.

mydatadoi <- paste0(dataverse_search(title = metatitle, type = "dataset")$global_id, '&version=DRAFT')


# Add the files to your dataset created above.

f <- lapply(files, function(x) add_dataset_file(file = x, dataset = mydatadoi))

What doesn't seem to work is creating a dataset using create_dataset instead of initiate_sword_dataset() before or after retrieving the service document:

ds <- create_dataset('UU', body = metadat)

Which produces:

Error in create_dataset("UU", body = metadat) : 
  Internal Server Error (HTTP 500).

And also the add_file produces an error when used instead of add_dataset_file():

f <- lapply(files, function(x) add_file(file = x, dataset = mydatadoi))

Producing the error:

(file = x, dataset = mydatadoi) : 
  Bad Request (HTTP 400). 

I resolved the previous error in the original post by first searching for the complete draft doi of the created dataset package, but it is still producing an 400 error.

Note in the second post that the curl commands work as intended.

[kuriwaki]

Thanks. Managed to reproduced this on my end.

[Danny-dK]

@kuriwaki Need a bit of further help.

I tried to get the native API curl example mentioned in https://guides.dataverse.org/en/5.3/api/native-api.html at the topic "Create a Dataset in a Dataverse" working in R. They mention that you "must" supply a json file containing the metadata fields to create a dataset in a dataverse. An example of a json file required is provided by them at https://guides.dataverse.org/en/5.3/_downloads/dataset-create-new-all-default-fields.json. The curl code is:

curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X POST "https://demo.dataverse.org/api/dataverses/root/datasets" --upload-file "dataset-create-new-all-default-fields.json"

Using https://curl.trillworks.com/#r and some fiddling about (not really understanding what I'm doing), I eventually got to a working one in R:

require(httr)

headers = c(
  `X-Dataverse-key` = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
)

metadat <- upload_file('D:/dataset-create-new-all-default-fields.json')

res <-httr::POST(url = "https://demo.dataverse.nl/api/dataverses/ddktestdeleteme/datasets", httr::add_headers(.headers=headers), body = metadat)

My question is 2-fold:

1. How would one create a json metadata body file from within R

2. What is the main difference between my R code that works with httr and the code within the dataverse package:

create_dataset <- function(dataverse, body, key = Sys.getenv("DATAVERSE_KEY"), server = Sys.getenv("DATAVERSE_SERVER"), ...) {
    dataverse <- dataverse_id(dataverse, key = key, server = server, ...)
    u <- paste0(api_url(server), "dataverses/", dataverse, "/datasets/")
    r <- httr::POST(u, httr::add_headers("X-Dataverse-key" = key), body = body, encode = "json", ...)
    httr::stop_for_status(r, task = httr::content(r)$message)
    httr::content(r)
}

[kuriwaki]

Unfortunately, I'm not the best person to ask httr/curl questions (don't know enough). But I can help point to others.

Regarding 1, pyDataverse may have thought about this issue more. I asked at gdcc/pyDataverse#116

Regarding 2, well,

library(dataverse)

server <- "demo.dataverse.nl"
dataverse <- "ddktestdeleteme"
identical(paste0(dataverse:::api_url(server), "dataverses/", dataverse, "/datasets/"), # taken from create_dataset()
          "https://demo.dataverse.nl/api/dataverses/ddktestdeleteme/datasets") # given in working code
#> [1] FALSE

# but
identical(paste0(dataverse:::api_url(server), "dataverses/", dataverse, "/datasets"), # remove last slash
          "https://demo.dataverse.nl/api/dataverses/ddktestdeleteme/datasets")
#> [1] TRUE

^{Created on 2021-03-03 by the reprex package (v0.3.0)}

So in create_dataset() there's only that (a) trailing slash and (b) encode = "json" that's different, right? Do you mind trying editing (a), (b), and both out of the function and seeing if that fixes it?

Having no easy reprex is inherent to these upload examples, so thanks for your patience..

[Danny-dK]

@kuriwaki No worries, I appreciate all the help I'm getting!

I'll start with the positive. The create_dataset() function of the dataverse package actually works as long as you use upload_file() of the httr package along with the json file. Don't know why I missed this. So this works:

library(dataverse)
library(rstudioapi)
library(httr)

Sys.setenv("DATAVERSE_SERVER" = "demo.dataverse.nl")
Sys.setenv("DATAVERSE_KEY" = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')

metadat <- upload_file('D:/dataset-create-new-all-default-fields.json')

create_dataset('ddktestdeleteme', body = metadat)

I tried changing the functions to add or remove the slash and encode = "json" both in the httr code and in a new function resembling the create_dataset() function. For simplicity's sake, I'll just show the create_dataset(), but the exact same messages occur with adjusting the httr code.

#Dataverse's create_dataset funtcion. Note the first line within the function is removed as that seemed to give errors 
# and didn't seem required. Removed: dataverse <- dataverse_id(dataverse, key = key, server = server, ...)


create_datasettest <- function(dataverse, body, key = Sys.getenv("DATAVERSE_KEY"), server = Sys.getenv("DATAVERSE_SERVER"), ...) {
   
    u <- paste0(dataverse:::api_url(server), "dataverses/", dataverse, "/datasets/")
    r <- httr::POST(u, httr::add_headers("X-Dataverse-key" = key), body = body, encode = "json", ...)
    httr::stop_for_status(r, task = httr::content(r)$message)
    httr::content(r)
}

server <- "demo.dataverse.nl"
dataverse <- "ddktestdeleteme"
key <- "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
metatitle <- paste0("thisistheRtestmult2_", format(Sys.time(), '%Y%m%d_%H%M'))
metacreator <- "Doe, John"
metadescription <- "AnRtestupload"

metadat <- list(
              title       = "sometitle",
              creator     = "Doe, John",
              description = "AnRtestupload")

create_datasettest(dataverse, body = metadat, key = key, server = server)

The trailing backslash doesn't seem to matter. Adding or removing it does not seem to change the outcome.
Keeping the encode = "json" gives the 500 error code.
Removing the encode = "json" gives the 400 code with the added info:

Bad Request (HTTP 400). Failed to Error parsing Json: Unexpected char 45 at (line no=1, column no=2, offset=1). 

Thus, it all works when using httr::upload_file(somejson.json) in either the raw httr code or in the dataverse funcion create_dataset().

The pydataverse documentation also seems to list that a json must be specified. Looking at https://pydataverse.readthedocs.io/en/latest/ at the create dataset section they specify a json as ds.json(). So I'm assuming for the create_dataset() to work as intended, there needs to be a way to give the "body" option a content in the form of a json object or file. My current way of listing the 'metadat' just doesn't seem to be recognized because of the format and / or the items specified and I don't know how to specify a json file. The examples the dataverse help documentation give do show what a json file looks like https://guides.dataverse.org/en/5.3/_downloads/dataset-finch1.json but I don't know how to do this efficiently in R. Adding just an empty json as per example https://stackoverflow.com/a/20109959 and write it to local disk as per example https://rfaqs.com/reading-and-writing-json-files-in-r/, and then use metadat <- upload_file(createdempty.json) and use that as the body, it also fails. So I'm assuming certain fields are required by Dataverse and certainly need to be in a certain order / format.

::EDIT::
A helper function as used in the initiate_sword_dataset() would be nice to build the metadata structure:

build_metadata: https://rdrr.io/cran/dataverse/src/R/build_metadata.R
referred in initiate_sword_dataset: https://github.com/IQSS/dataverse-client-r/blob/master/R/SWORD_dataset.R

[Danny-dK]

Well...it's dumb as hell and ain't pretty, but it works. Using the minimal fields required for the json:

jsonmetaverse <- function(title, author, affiliation, description){
  headerjson <- paste0('{"datasetVersion": {"metadataBlocks": {"citation": {"fields": [{')
  titlejson <- paste0('"value":"', title, '", "typeClass": "primitive", "multiple": false,"typeName": "title"},')
  authornamejson <- paste0('{"value": [{"authorName": {"value":"', author, '", "typeClass": "primitive", "multiple": false,"typeName": "authorName"},')
  authoraffiliationjson <- paste0('"authorAffiliation": {"value":"', affiliation, '", "typeClass": "primitive", "multiple": false, "typeName": "authorAffiliation"}}],')
  authorjson <- paste0(authornamejson, authoraffiliationjson, '"typeClass": "compound", "multiple": true,"typeName": "author"},')
  descriptionvaluejson <- paste0('{"value": [ {"dsDescriptionValue":{"value":"', description, '", "multiple":false, "typeClass": "primitive", "typeName": "dsDescriptionValue"}}],')
  descriptionfulljson <- paste0(descriptionvaluejson, '"typeClass": "compound","multiple": true,"typeName": "dsDescription"}')
  closingjson <- paste0(']}}}}}')
  fulljson <-paste0(headerjson, titlejson, authorjson, descriptionfulljson, closingjson)
}

testmeta <- jsonmetaverse(title = 'testing stuff', author= 'Doe John', affiliation = 'testverse', 
                          description = 'testing working json upload meta')



library(dataverse)

Sys.setenv("DATAVERSE_SERVER" = "demo.dataverse.nl")
Sys.setenv("DATAVERSE_KEY" = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx')

create_dataset('ddktestdeleteme', body = testmeta)

Note that you can actually create a dataset with an empty json file as mentioned at IQSS/dataverse#6752, which I managed to do, but then the dataset cannot be opened or viewed within dataverse.

[kuriwaki]

Thank you @Danny-dK for the investigation!

I am still learning the create* parts of the package, but it seems like you're post highlights 2 concrete and relatively straightfoward things:

1. Either document that the body argument of create_dataset must be a httr::upload_file object, OR (perhaps better) write in httr::upload_file into the function
2. Implement dataverse::build_metadata into create_dataverse too, just like initiate_sword_dataset, so that users can input a R list.

Would you be comfortable or interested in starting a fork/branch to make and test these fixes?

In the mid/long-term, I am trying to figure out if we really need to maintain a native vs. sword API, two parallel things, to do the same thing.