[WIP] Adding query widget functionality to export

[ironmaniiith]

Description (will update according to progress made): The current export functionality simply exports everything that is present in the site and doesn't give the flexibility to choose the content of interest that needs to be exported. For this, a query widget is added, using which a user can use it to filter out the contents which he/she is interested in and can export only that limited content.

Currently, the content of this PR is accessible at @@export-form url. Upon completion, this will be moved to the main export section of @@import-export and the existing code will be modified accordingly

Update: The content is already moved to @@import-export

[djay]

• The form isn't a control panel form.
• There is an existing form so I'm not sure why you need to create a new one.
• the PR doesn't explain the scope of the task you are doing. Export already exists.
• The fields in this form are from collections and don't relate to export

[ironmaniiith]

• There is an existing form so I'm not sure why you need to create a new one.

Which one?

• the PR doesn't explain the scope of the task you are doing. Export already exists.

I'll add the details of this PR

• The fields in this form are from collections and don't relate to export

Yeah, this was just a basic working code that I pushed. I'll modify it. As my current priority, I'm looking at how to make the zip file download thing working

• The form isn't a control panel form.

There were too many forms that I've seen from several libraries, not sure which form exactly to use here so I implemented it like this. I don't know which form you are refering to which already exists? Also in the currrent implementation of import-export page, a template is used which however we dont have to use here.

Questions:

1. If I want to add a template which still let me get the data of the form, how do I do that?
2. Along with the form, if I want to add the checkboxes for metadata, how to do that?
3. If instead of returning from the current button handler function, if we can return from a method of a class whose parent is BrowserView class, how to do that. This I'm asking because it will make the zip download thing easier

[ironmaniiith]

I just checked the contents of the zip file. Seems like I'm able to export the correct contents by using setBody (here), however, I can't have the actual zip contents in the body, doing which leads to an error related to encoding decoding. If I convert it to base64 encoded contents, the zip that gets exported is also simply a file with the same base64 content (and not the actual zip content)
I tried opening the zip file, decoding it from base64 and saved it locally. It gave me the correct zip file.

[ironmaniiith]

The current commit resolves all the errors related to zip file downloading and currently we have a working portal which is able to export the contents which are selected as part of the query widget.
Next, I'll be adding the functionality to chose which metadata to export.

Here is a small demo video for the working functionality: https://streamable.com/crpw8

Alternative link

[Shriyanshagro]

@ironmaniiith that's very well done.
This should be able to resolve a lot of stories which include user queries to filter data.
However, in your uploaded video, it looks like, all required fields in CSV is not generated. Solve that before the merge of this PR

[ironmaniiith]

@ironmaniiith that's very well done.

Thanks @Shriyanshagro

This should be able to resolve a lot of stories which include user queries to filter data.

Yeah, I think this will cover all the stories that were there in the doc related to export.

However, in your uploaded video, it looks like, all required fields in CSV is not generated. Solve that before the merge of this PR

I was actually just experimenting with the query widget for just one field in the CSV. The main target at the time was to get the working code which will be able to cover the majority of the stories. Also I was trying to resolve the other issues which were the major blocker previously. I'll add that part in. I infact had that already in my TODO list

[ironmaniiith]

Work done in this PR:

• Added query widget for performing selection of the content of interest (previously this functionality was not there and we could only export whole content present in the site).

• Extended the functionality of export for exporting csv file of selected metadata (This was there previously as well, however it again exported the metadata of all the files, but with current implementation, this is not an issue)

• Extended the functionality of export by providing export type (csv, files only or combined)

• Added an extra feature of exporting the content without preserving the files's relative location (will cover this use case with an example in the story following this paragraph)

• Added a mechanism to reset the current selection

• Added comments, notes and TODOs wherever required

• Performed error handling wherever needed (if there are any more use cases that anyone can think of please do let me know)

• Extensively used existing functionalites in plone (unlike the previous approach, where templates and js were written manually). Same functionality is achieved in much shorter and better way

• One more feature that was added was to display the metadata in sorted order. Many a times finding a particular metadata which a user is interested in exporting is difficult because of no ordering (and he/she simply has to look at all the options). With the sorted metadata, it becomes much easier to locate any particular metadata and hence improves the user experience.

• Made demo videos to go through majority of the functionality implemented in this PR (link at the end)

Improvements done while completing this PR

• Figured out how to use QueryBuilder in order to obtain the results of the query made using QueryStringFieldWidget. QueryStringFieldWidget doesn't provide the results set by default. This was done by going through the library code of QueryBuilder which also helped developing a better understanding of plone architecture and workflow.

• Removed the code that was performing redirection eariler by sending the contents of the query json in the url. The main reason due to which it was done was the inability to export zip file, which caused decoding error upon download. This was resolved by using response.write instead of response.setBody method

• Followed better coding conventions for methods name

• Performed error handling

• Modified existing codebase (mainly utils.py) to support new functionalities added in this PR

• While adding the metadata part in export, I realized that we can't directly hardcode the metadata in the form scehma (both because it's not constant and also it's not a good practise). For solving this thing, in order to extend the metadata values with the form schema, dynamic vocabularies was used.

Note

• Previous implementation always exported some MUST_INCLUDED_ATTRIBUTES which is not done in this work. However, it maybe required to export those attributes as they will be later required in future. I see 2 ways of accomplishing this.

  1. Either we stick to the previous approach and export MUST_INCLUDED_ATTRIBUTES no matter the user preference.
  2. We can ask user whether he/she wants to export those attributes by adding a field in the form

• A case may arise when there are more than 1 file with the same name and user still wishes to export the content without preserving these files' relative path. How to handle this case is open to our perception and can be added later if required. (some possible solutions are: we can infact leave the implementation as it is, detect if such a case arises and can warn user of such behaviour, or we can rename the content by appending increasing numbers in front of them etc.). Mentor suggestion is required here.

• Another trivial note to make here is when user selects to export only csv, then preserve relative path does not have any effect (as no files are exported)

• A problem that has been extensively discussed in the past still remains open. What do we do in case of a very big zip file which doesn't even fit in the memory. This is something to be worked upon for sure later.

• Adding the metadata functionality required getting all the fields of metadata and to have them in form schema. Hardcoding the metadata in form schema was also not an option (both because the fields metadata are not constant and it's a bad practise). This particular part was not direct and required me to use dynamic vocabularies.

Video links:

• export-combined-zip-without-preserving-path
• export-csv-only
• export-files-only-without-preserving-path
• reset-feature