Openapi by jotelha · Pull Request #1 · ljyanesm/dtool-lookup-server

jotelha · 2022-04-12T14:06:52Z

Flask < 2.0 causes conflict with current flask-smorest requirement >= 2.0, see

$ python --version
Python 3.8.10
$ cat requirements.in 
alembic
gunicorn
pymongo
psycopg2
importlib_metadata
dtool
dtool-ecs
dtool-irods
dtool-smb
dtool-s3
dtool-lookup-server
flask
flask-jwt-extended[asymmetric_crypto]>=4.0
flask-sqlalchemy
flask-migrate
flask-pymongo
flask-marshmallow
flask-smorest
flask-cors
markupsafe==2.0.1
marshmallow-sqlalchemy
pyyaml
$ pip-compile requirements.in 
Could not find a version that matches flask<2.0,<3.0,>=0.10,>=0.11,>=0.9,>=1.0,>=2.0 (from -r requirements.in (line 12))
Tried: 0.1, 0.2, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.5.2, 0.6, 0.6.1, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.9, 0.10, 0.10.1, 0.11, 0.11, 0.11.1, 0.11.1, 0.12, 0.12, 0.12.1, 0.12.1, 0.12.2, 0.12.2, 0.12.3, 0.12.3, 0.12.4, 0.12.4, 0.12.5, 0.12.5, 1.0, 1.0, 1.0.1, 1.0.1, 1.0.2, 1.0.2, 1.0.3, 1.0.3, 1.0.4, 1.0.4, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.1.3, 1.1.3, 1.1.4, 1.1.4, 2.0.0, 2.0.0, 2.0.1, 2.0.1, 2.0.2, 2.0.2, 2.0.3, 2.0.3, 2.1.0, 2.1.0, 2.1.1, 2.1.1
Skipped pre-versions: 2.0.0rc1, 2.0.0rc1, 2.0.0rc2, 2.0.0rc2
There are incompatible versions in the resolved dependencies:
  flask (from -r requirements.in (line 12))
  Flask>=0.9 (from flask-cors==3.0.10->-r requirements.in (line 19))
  Flask<3.0,>=1.0 (from flask-jwt-extended[asymmetric_crypto]==4.3.1->-r requirements.in (line 13))
  Flask>=0.9 (from flask-migrate==3.1.0->-r requirements.in (line 15))
  Flask (from flask-marshmallow==0.14.0->-r requirements.in (line 17))
  Flask>=0.11 (from flask-pymongo==2.3.0->-r requirements.in (line 16))
  flask<3,>=2.0 (from flask-smorest==0.37.0->-r requirements.in (line 18))
  Flask>=0.10 (from flask-sqlalchemy==2.5.1->-r requirements.in (line 14))
  flask<2.0 (from dtool-lookup-server==0.17.2->-r requirements.in (line 11))

marshmallow-sqlalchemy is necessary when using SQLAlchemy. Also, I think the server core should not depend on specific storage brokers like "dtool_irods", "dtool_s3", "dtool_ecs".

… openapi

…rver into openapi

…a in sql_models.py and schmemas.py caused troubles for autogenerated Python client API. For now, resorted to suboptimal suffix "SQLAlchemy" for the schemata defined within sql_models.py. Do we need actually need the different schema in sql_models.BaseURISQLAlchemySchema(ma.SQLAlchemyAutoSchema) and schemas.class BaseURISchema(Schema) ? * sql_models.Dataset(db.Model) defines columns frozen_at and created_at as db.DateTime(), its method as_dict() converts those with dtoolcore.utils.timestamp(self.frozen_at). This, however, results in the autogenerated client API to expect a serialized datetime object and throw an exception when receiving float. For now, inverted the previos behavior of the server to always return timestamps as floats to returning datetime objects instead. What should be the way here in the future? * Standardized spelling of Uri to URI in models and schema for all occurences.

jotelha · 2022-04-21T17:33:28Z

A few remarks here:

When working within these openapi boundaries, what's not clear to me is how to handle arbitrarily structured data? I.e. for the config of the server, that's a key-value dict, https://github.com/jic-dtool/dtool-lookup-server/blob/master/dtool_lookup_server/config.py, that would also present plugin-specific configs nested below toplevel, again with possibly arbitrarily nested objects. The schema to capture this would clearly just be the general JSON or yaml specifications, and that's the case for a dataset's README.yml as well (if it's not just handled as a string). Clearly, one does not want to prescribe a narrow schema here, but still wants that reflected somehow in the documentation. Are empty schemas like those https://github.com/jotelha/dtool-lookup-server/blob/eca0305dbe1ed1575e51f2b5367e72dbca6b1cab/dtool_lookup_server/schemas.py#L87-L92 legitimate?
Removed @bp.response(200, Dict)
Standardized spelling of Uri to URI in models and schema for all occurrences.
Different cases in the URI acronym like BaseURI, BaseUri, BaseURISchema, and BaseUriSchema in sql_models.py and schemas.py caused troubles for autogenerated Python client API. For now, resorted to suffix SQLAlchemy for the schema defined within sql_models.py due to a lack of imagination. Do we actually need the different schema in sql_models.BaseURISQLAlchemySchema(ma.SQLAlchemyAutoSchema) and schemas.BaseURISchema(Schema) ?
sql_models.Dataset(db.Model) defines columns frozen_at and created_at as db.DateTime(), its method as_dict() converts those with dtoolcore.utils.timestamp(self.frozen_at). This, however, results in the autogenerated client API to expect a serialized datetime object and to throw an exception when receiving float. For now, inverted the previous behavior of the server to always return timestamps as floats to returning datetime objects instead, but that's only been done as a simple way to test against the autogenerated client. What should be the way here in the future?

jotelha · 2022-04-21T22:20:21Z

Treat created_at and frozen_at as floats in schema with
https://github.com/jotelha/dtool-lookup-server/blob/9d1b8cc751f1d1e143bae10d9454d86be438dc13/dtool_lookup_server/sql_models.py#L122-L128 as suggested below https://marshmallow.readthedocs.io/en/latest/quickstart.html#implicit-field-creation. Not sure whether this is legitimate.

jotelha · 2022-04-21T22:21:32Z

Currently, tests on register roue will fail with

ERROR    dtool_lookup_server:app.py:1455 Exception on /dataset/register [POST]
Traceback (most recent call last):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/webargs/core.py", line 452, in wrapper
    return func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/arguments.py", line 82, in wrapper
    return func(*f_args, **f_kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_smorest/response.py", line 90, in wrapper
    func(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 127, in decorator
    return current_app.ensure_sync(fn)(*args, **kwargs)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/dataset_routes.py", line 148, in register
    dataset_uri = register_dataset(dataset_info)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 686, in register_dataset
    register_dataset_descriptive_metadata(dataset_info.copy())
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 632, in register_dataset_descriptive_metadata
    _register_dataset_descriptive_metadata(collection, dataset_info)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/dtool_lookup_server/utils.py", line 658, in _register_dataset_descriptive_metadata
    exists = collection.find_one(query)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/collection.py", line 1419, in find_one
    for result in cursor.limit(-1):
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1248, in next
    if len(self.__data) or self._refresh():
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
    self.__send_message(q)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/cursor.py", line 1052, in __send_message
    response = client._run_operation(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1267, in _run_operation
    return self._retryable_read(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1371, in _retryable_read
    return func(session, server, sock_info, read_pref)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1263, in _cmd
    return server.run_operation(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/server.py", line 100, in run_operation
    message = operation.get_message(read_preference, sock_info, use_cmd)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 375, in get_message
    request_id, msg, size, _ = _op_msg(
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/pymongo/message.py", line 673, in _op_msg
    return _op_msg_uncompressed(flags, command, identifier, docs, opts)
  File "/mnt/dat2/git/dtool/dtool-lookup-server/venv/lib/python3.8/site-packages/bson/binary.py", line 267, in from_uuid
    raise ValueError(
ValueError: cannot encode native uuid.UUID with UuidRepresentation.UNSPECIFIED. UUIDs can be manually converted to bson.Binary instances using bson.Binary.from_uuid() or a different UuidRepresentation can be configured. See the documentation for UuidRepresentation for more information.

jotelha · 2022-04-21T23:04:55Z

Treat UUIDs as string for now, keep https://pymongo.readthedocs.io/en/stable/examples/uuid.html#handling-uuid-data in mind.

WIP - OpenAPI support

Dataset readme files tend to be in YAML format. However, this is a convention rather than a prescription. When generating the information to upload a dataset to the lookup server the `utils.generate_dataset_info` converted the YAML text to a dictionary and it was registered as such in the mongo database. This would result in errors if the readme contained invalid YAML. It would be better if the base system deal with the readme as a string of text rather than trying to interpret it.

jotelha · 2022-09-21T13:09:53Z

Reopened at jic-dtool#27

jotelha and others added 16 commits April 12, 2022 11:30

MAINT: Removed conflicting dependencies from setup.py

dbea4a0

MAINT: Needs marshmallow-sqlalchemy

6ee07f2

MAINT: server should not depend explicitly on dtool_s3

0e0a2c0

Merge branch 'ljyanesm:openapi' into openapi

775b479

MAINT: launch script for quick testing

d6c8083

Merge branch 'openapi' of github.com:jotelha/dtool-lookup-server into…

47878b4

… openapi

MAINT: exclude auto-generated keys and pycharmm .idea folder

980cb16

typo

3e6fdfa

Merge branch 'openapi' of https://github.com/ljyanesm/dtool-lookup-se…

a3fe388

…rver into openapi

MAINT: devel requirements.txt for quick testing setup

3802e57

MAINT: AnnotationsSchema

a64a661

DOC: development environment and openapi-generator-cli examples

70289aa

MAINT: empty ConfigSchema

f86de83

DOC: Working auto-generated openapi client

abf62ea

DOC: three APIs

f61ba5a

jotelha added 3 commits April 21, 2022 21:11

MAINT: Removed silly SQLAlchemy suffix from scheme where not necessary

5b9a59d

DOC: export token

8d5b131

MAINT: changed back to timestamps as floats in dataset schema

9d1b8cc

MAINT: Treat UUIDs as String in schema

31e71b6

jotelha and others added 6 commits April 22, 2022 14:49

MAINT: Specify remote host and token generator in env vars

c5bdfea

MAINT: aiopenapi3 client sample

b509c3e

Merge pull request jic-dtool#24 from ljyanesm/openapi

5638bd3

WIP - OpenAPI support

Fix dependencies

1034626

Add missing fields to RegisterDatasetSchema following merge to master

99db7db

Fix issue with test harness identified by new schema

ead0141

tjelvar-olsson and others added 5 commits June 25, 2022 19:22

Fix broken import

d7c5fc1

Merge branch 'yanes-openapi'

0de6f54

MAINT: removed all devel relics

dcfadfa

MAINT: removed redundant requirement from setup.py

e900829

jotelha mentioned this pull request Sep 21, 2022

Bits and pieces on openapi implementation jic-dtool/dservercore#27

Closed

jotelha closed this Sep 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Openapi#1

Openapi#1
jotelha wants to merge 31 commits intoljyanesm:openapifrom
jotelha:openapi

jotelha commented Apr 12, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Sep 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jotelha commented Apr 12, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Apr 21, 2022

Uh oh!

jotelha commented Sep 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants