Getting started with dpm

[cboettig]

Advice from the manual directs me to edit setup.py after creating a new project repository with init, however I see no template setup.py in the directory it creates. I do see a datapackage.json file, perhaps this has replaced setup.py as the correct place to edit metadata? Is some metadata required and other data optional?

I have added my API key to the configuration file as directed, but on attempting to register an example data package I get the error:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/dpm-0.10-py2.7.egg/dpm/cli/base.py", line 172, in main
    self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/dpm-0.10-py2.7.egg/dpm/cli/standard.py", line 231, in run
    index_to.register(pkg)
  File "/usr/local/lib/python2.7/dist-packages/dpm-0.10-py2.7.egg/dpm/index/ckan.py", line 88, in register
    self.ckan.package_register_post(package_dict)
  File "/usr/local/lib/python2.7/dist-packages/ckanclient-0.9-py2.7.egg/ckanclient/__init__.py", line 374, in package_register_post
    self.open_url(url, data, headers)
  File "/usr/local/lib/python2.7/dist-packages/ckanclient-0.9-py2.7.egg/ckanclient/__init__.py", line 346, in open_url
    raise CkanApiConflictError(self.last_status)
CkanApiConflictError: 409

Not sure what I have done wrong, any advice?

Finally, let me know if there is a user mailing list, forum, or otherwise preferable place to post my elementary issues as I learn dpm or if it is okay to post them here. Thanks!

[rufuspollock]

409 conflict indicates that package/dataset is already registered (you may need to use update instead of register). Which CKAN site are you trying this out on?

[cboettig]

I'm trying with the defaults, http://thedatahub.org/api
I created an account on thedatahub.org and obtained my api key, then just ran dpm setup config and dpm init WrightscapeDataPackage.

[cboettig]

I tried another example, editing more of the metatdata, and now I get a different error when attempting to register:

    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

1. What is the minimal metadata required?
2. Does this just involve filling in the blanks in the datapackage.json file?
3. Do I need to add or edit anything else to get the commit to work?

Thanks for the help, this looks like an excellent project and with a solid API I hope I can incorporate data deposition into thedatahub.org as part of my daily workflow. I just need to get the basics down first.

[rufuspollock]

@cboettig hmmm. So you don't have any datasets registered on the DataHub yet: http://thedatahub.org/user/cboettig

409 conflict may have been because your API key is not set. Do you have in your ~/.dpmrc file

[index:ckan]
ckan.url = http://thedatahub.org/api/
ckan.api_key = {your-api-key}

I've also spotted a bug in the manual regarding editing setup.py which I have corrected (you do not edit setup.py - you edit datapackage.json in the latest (v0.10) version of dpm -- though it sounds you had already worked that out).

To answer specific questions:

1. Minimal metadata is the name which is set by default from the dataset/package name.
2. Yes just fill in blanks in datapackage.json but do ensure this remains a valid JSON file (this is annoying and we are working to fix this). Your error looks like a problem opening the datapackage.json. I suggest running with -v option (see below)
3. No you do not need to add anything else.

Run commands with -v option to get more debug output e.g. dpm -v register . ckan://

[cboettig]

Great, got things to register with a blank package. I think the problem was I had capital letters in the package name?

Now I have problems uploading. First, I'm not clear how uploading works. From the manual, it suggests copying data into the registered directory. I see the package has a subdirectory called data -- should the data go in that directory? The instructions seem to put it in the base folder:

$ cp mydata.csv my-new-dataset
$ cp mydata.js my-new-dataset
$ etc ...

Then to upload, it says to use

$ dpm upload {your-file} ckan://{yourpackagename}/{filename}

I find this confusing. Is {your-file} the path to the file within the package, or just anywhere on the local disk? Why have we copied it to the directory if we need to specify it's path again anyhow?

The command fails for me, when giving it a simple README.md file to add:

dpm upload README.md ckan://wrightscapedata/README.md
Uploading, please be patient ...
Error: No suitable OFS backend for uploading found

[** For (lots) more information run with --debug **]

[rufuspollock]

Upload just does not work at the moment -- sorry that should be disabled in dpm temporarily. I'd also wonder what you are uploading -- if it is raw data it may be better to put that in the data store via the data API rather than uploading the file itself.

I'm sorry about this: you've caught us at a moment where dpm has lagged development of the datahub/CKAN :-)

More info about plans here in this document: https://github.com/okfn/dpm/blob/master/doc/new-plan-2011-nov.rst

Meanwhile if you want to upload to the DataStore check out https://github.com/okfn/datastore-client (and yes, we plan to wire this into dpm as soon as we can -- it would be part of the push command ...)

[rufuspollock]

@cboettig I'm working to get upload fixed today or tomorrow. In the meantime any feedback or suggestions from your experience so far would be very welcome.