diff --git a/.abi-check/6.29.1_arenadata68/postgres.symbols.ignore b/.abi-check/6.29.1_arenadata68/postgres.symbols.ignore new file mode 100644 index 000000000000..aa4c33e2be25 --- /dev/null +++ b/.abi-check/6.29.1_arenadata68/postgres.symbols.ignore @@ -0,0 +1 @@ +ConfigureNamesBool_gp diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md index c90a6f5a1253..5eaf063d68d9 100644 --- a/.github/CODE_OF_CONDUCT.md +++ b/.github/CODE_OF_CONDUCT.md @@ -1,50 +1,45 @@ -# Code of Conduct +# Greengage DB Community Code of Conduct -## Intro +Greengage DB is an open community project. Every Greengage DB user or developer is responsible for making the community a place where people want to come back. We expect all community members to maintain professional ethics and communicate respectfully and politely. -Standing on the shoulders of the giants means not only leveraging the code base of the most successful open source relational database (PostgreSQL) but also taking a page out of one of the most successful open source governance bodies: Apache Software Foundation (ASF). Greengage Database developer community has adopted not only the Apache License but also the following code of conduct heavily borrowing from an ASF’s: +## Community Development Principles -This code of conduct applies to all spaces that are associated with the participation in the Greengage Database open source project, including chat, all public and private mailing lists, issue trackers, wikis, blogs, Twitter, and any other communication channel used by our community. A code of conduct which is specific to in-person events (ie., conferences, meetups, etc.) is expected to be a superset of this document covering additional principles put forward by the organizers of the event(s) and landlords of the space where the event is held. +In developing code, documentation, and any other products of intellectual work, we are guided by principles that are important for the development of the community. -We expect this code of conduct to be honored by everyone who participates in the Greengage Database community formally or informally, or claims any affiliation with the project, in any activities and especially when representing the Greengage Database project, in any role. +**Collaboration** -This code is not exhaustive or complete. It serves to distil our common understanding of a collaborative, shared environment and goals. We expect it to be followed in spirit as much as in the letter so that it can enrich all of us and the technical communities in which we participate. +Anyone can join the Greengage DB community and make a contribution. -## Specific guidelines +**Equality** -We strive to: -1. **Be open** We invite anyone to participate in our community. We preferably use public methods of communication for project-related messages, unless discussing something sensitive. This applies to messages for help or project-related support, too; not only is a public support request much more likely to result in an answer to a question, it also makes sure that any inadvertent mistakes made by people answering will be more easily detected and corrected. -2. **Be empathetic**, welcoming, friendly, and patient. We work together to resolve conflict, assume good intentions, and do our best to act in an empathetic fashion. We may all experience some frustration from time to time, but we do not allow frustration to turn into a personal attack. A community where people feel uncomfortable or threatened is not a productive one. We should be respectful when dealing with other community members as well as with people outside our community. -3. **Be collaborative**. Our work will be used by other people, and in turn will depend on the work of others. When we make something for the benefit of the project, we are willing to explain to others how it works, so that they can build on the work to make it even better. Any decision we make will affect users and colleagues, and we take those consequences seriously when making decisions. -4. **Be inquisitive**. Nobody knows everything! Asking questions early avoids many problems later, so questions are encouraged, though they may be directed to the appropriate forum. Those who are asked should be responsive and helpful, within the context of our shared goal of improving Greengage Database project code. -5. **Be careful in the words that we choose**. Whether we are participating as professionals or volunteers, we value professionalism in all interactions, and take responsibility for our own speech. Be kind to others. Do not insult or put down other participants. Harassment and other exclusionary behaviour are not acceptable. This includes, but is not limited to: - * Violent threats or language directed against another person. - * Sexist, racist, or otherwise discriminatory jokes and language. - * Posting sexually explicit or violent material. - * Posting (or threatening to post) other people's personally identifying information ("doxing"). - * Sharing private content, such as emails sent privately or non-publicly, or unlogged forums such as IRC channel history. - * Personal insults, especially those using racist or sexist terms. - * Unwelcome sexual attention. - * Excessive or unnecessary profanity. - * Repeated harassment of others. In general, if someone asks you to stop, then stop. - * Advocating for, or encouraging, any of the above behaviour. -6. **Be concise**. Keep in mind that what you write once will be read by hundreds of persons. Writing a short email means people can understand the conversation as efficiently as possible. Short emails should always strive to be empathetic, welcoming, friendly and patient. When a long explanation is necessary, consider adding a summary. - Try to bring new ideas to a conversation so that each mail adds something unique to the thread, keeping in mind that the rest of the thread still contains the other messages with arguments that have already been made. - Try to stay on topic, especially in discussions that are already fairly large. -7. **Step down considerately**. Members of every project come and go. When somebody leaves or disengages from the project they should tell people they are leaving and take the proper steps to ensure that others can pick up where they left off. In doing so, they should remain respectful of those who continue to participate in the project and should not misrepresent the project's goals or achievements. Likewise, community members should respect any individual's choice to leave the project. +The contribution of each community member will be assessed equally, based on its objective significance for the project. -## Diversity statement +**Opportunities** -Greengage Database project welcomes and encourages participation by everyone. We are committed to being a community that everyone feels good about joining. Although we may not be able to satisfy everyone, we will always work to treat everyone well. +Community members who contribute to Greengage DB can influence the direction of the product's development. -No matter how you identify yourself or how others perceive you: we welcome you. Though no list can hope to be comprehensive, we explicitly honour diversity in: age, culture, ethnicity, genotype, gender identity or expression, language, national origin, neurotype, phenotype, political beliefs, profession, race, religion, sexual orientation, socioeconomic status, subculture and technical ability. +**Availability** -Though we welcome people fluent in all languages, Greengage Database project development is conducted in English. +The project code will be distributed under an open license for community members anywhere in the world. -Standards for behaviour in this community are detailed in the Code of Conduct above. We expect participants in our community to meet these standards in all their interactions and to help others to do so as well. +## Code of Conduct in Action -## Reporting guidelines +This code of conduct applies to all available channels of interaction between community members, including comments on code or documentation, the project's social media, and online and offline events that are partially or fully dedicated to Greengage DB. -While this code of conduct should be adhered to by participants, we recognize that sometimes people may have a bad day, or be unaware of some of the guidelines in this code of conduct. When that happens, you may reply to them and point out this code of conduct. Such messages may be in public or in private, whatever is most appropriate. However, regardless of whether the message is public or not, it should still adhere to the relevant parts of this code of conduct; in particular, it should not be abusive or disrespectful. +Every member of the Greengage DB community is expected to adhere to the project’s code of conduct, especially when speaking on behalf of Greengage DB in the public space. +We encourage everyone to be interested in Greengage DB, regardless of their track record or formal qualifications. This does not negate the important procedures for code and documentation quality, but it does require respectful and polite treatment. -If you believe someone is violating this code of conduct, you may reply to them and point out this code of conduct. Assume good faith; it is more likely that participants are unaware of their bad behaviour than that they intentionally try to degrade the quality of the discussion. Should there be difficulties in dealing with the situation, you may report your compliance issues in confidence to coc@greengagedb.org. This will go to an individual who is entrusted with your report. +The Greengage DB project exists thanks to the voluntary contributions of its members. Some of them may leave the project over time, and this decision should be respected in the same way as their initial involvement. We expect this code of conduct to be followed in the event of parting with the project, and the departing community member is supposed to treat those who remain with respect. + +## Appropriate Conduct + +No personal characteristics can serve as grounds for refusing to work for the benefit of the Greengage DB project or belittling your contribution to it. This code of conduct considers any attacks on a person unacceptable. These are the basic principles of communication in the community. + +Unacceptable behavior is any threat to use violence, damage social or professional reputation, expressed to an individual or a group of people. Threats to damage the Greengage DB project or its reputation or the implementation of such threats are also a violation of this code of conduct. Harassment of any kind and violation of the basic principles of communication are not tolerated in the Greengage DB community. + +In the event of receiving a warning about inappropriate behavior, it is necessary to immediately stop it. Subsequent violations of the Code of Conduct will serve as grounds for taking restrictive measures against the violator, up to and including exclusion from the community. + +## Reporting + +In many cases, simply sending a link to this code of conduct will be enough to maintain a good community culture. If you see a discussion where the code may potentially be violated, proactively send a link to it. +In some cases, violations may require issuing warnings and other restrictive measures. In this case, please contact us at **code@greengagedb.org**. diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml new file mode 100644 index 000000000000..aca16a9294a0 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -0,0 +1,84 @@ +name: Bug Report +description: Create a report to help us improve GreengageDB +title: "[Bug] " +labels: ["bug"] +assignees: [] + +body: + - type: textarea + id: problem_description + attributes: + label: "Describe the problem" + description: "Please describe the issue you observed, and any steps we can take to reproduce it" + placeholder: "Describe the problem you encountered..." + validations: + required: true + + - type: textarea + id: reproduction_steps + attributes: + label: "How to reproduce" + description: "Steps to reproduce the behavior" + placeholder: | + 1. + 2. + 3. + ... + validations: + required: true + + - type: textarea + id: expected_behavior + attributes: + label: "Expected behavior" + description: "A clear and concise description of what you expected to happen" + placeholder: "Describe what you expected to happen..." + validations: + required: true + + - type: textarea + id: additional_data + attributes: + label: "Additional data" + description: | + If the problem is SQL-related, include a copy of the SQL query and the schema. + + If a segment in your cluster encountered a fatal error, supply the contents of the log directory (at minimum of the affected segment(s), but preferably all segments). + + Note that log files can contain confidential information. + placeholder: | + SQL Query: + Schema: + Log files: + ... + + - type: input + id: environment_greengagedb + attributes: + label: "GreengageDB version" + description: "What version of GreengageDB are you using?" + validations: + required: true + + - type: input + id: environment_os + attributes: + label: "Server OS" + description: "What operating system is the server running on?" + placeholder: "e.g. Linux/Distrib version" + validations: + required: true + + - type: input + id: environment_client + attributes: + label: "Client application" + description: "What client application are you using?" + placeholder: "e.g. psql, JDBC, pxf, ..." + + - type: textarea + id: additional_context + attributes: + label: "Additional context" + description: "What was the impact? Add any other context about the problem here" + placeholder: "Add any other relevant context about the problem..." \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 000000000000..f61d70e28ade --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,17 @@ +--- +name: Feature request +about: Suggest an idea for Greengage project +title: "[Feature]" +labels: '' +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. + +**Describe the solution** +A clear and concise description of what you want to happen. + +**Additional context** +Add any other context about the feature request, user stories or examples of usage. diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 86450215b42a..08d466d88685 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -37,6 +37,82 @@ operating systems: DockerHub. Runs for push to `main` (retags to `latest`) and tags (uses tag like `6.28.2`) after build. +## Release Workflow + +A separate workflow `Greengage release` handles the uploading of Debian package +to GitHub releases. It is triggered when a release is published and uses a +composite action to manage package deployment. + +### Key Features + +- **Triggers:** `release: [published]` - Runs when a release is published, +including re-publishing. +- **Concurrency:** Uses the same concurrency group as the CI workflow +(`Greengage CI-${{ github.ref }}`) to ensure proper sequencing and prevent race +conditions. +- **Cache-based Artifacts:** Restores built packages from cache using the +commit SHA as the key, rather than downloading artifacts from previous jobs. +- **Manual Recovery:** If the cache is missing, the workflow checks the status +of the last build for the tag and provides clear instructions for manual +intervention. It does not automatically trigger builds to avoid infinite loops. +- **Safe Uploads:** Uploads packages with fixed naming patterns and optional +overwrite (`clobber` flag). + +### Behavior + +1. **Normal Flow (Cache Available):** Restores packages from cache, renames +them to the pattern `${PACKAGE_NAME}${VERSION}.${EXT}`, and uploads to the +release. +2. **Cache Miss Scenarios:** + - **No previous build or previous build successful:** Provides instructions + to manually trigger the CI build, then restart the release workflow. + - **Previous build failed:** Reports the failure with a link to the failed + run and requires manual fixing before retrying. + +The release workflow is designed to be robust and provide clear feedback when +issues occur, ensuring that releases are always consistent and reliable. + +## SQL Dump Workflow + +A separate workflow `Greengage SQL Dump` is responsible for generating SQL dump +artifacts after the main CI process completes successfully. It is triggered +automatically upon the completion of the `Greengage CI` workflow. + +### Key Features + +- **Triggers:** `workflow_run: workflows: ["Greengage CI"], types: [completed]` +- **Branch Targeting:** Runs only for the `main` and `7.x` branches. +- **Version Detection:** Automatically determines the database version (6 or 7) +based on the triggering branch. +- **Artifact Creation:** Executes regression tests with the `dump_db: "true"` +parameter to generate a SQL dump archive, which is then uploaded as a workflow +artifact. +- **Controlled Execution:** Since the main CI workflow runs on `main` and `7.x` +branches only for push events (which occur after final merge of a PR), SQL dump +are generated exclusively for verified, approved patches after they are merged +into the main branches. +- **Artifact Retention:** The generated SQL dump artifact is retained 90 days +after the last download. Each new run of the `behave tests gpexpand` workflow +(which consumes this artifact as a consumer) resets this retention period to +90 days when it downloads the artifact. + +### Behavior + +1. **Triggering:** Automatically starts after the `Greengage CI` workflow +finishes on the `main` or `7.x` branch. +2. **Preparation:** Configures Docker storage on the runner to utilize +`/mnt/docker` for increased disk space. +3. **Version Mapping:** Maps the branch name (`main` -> version 6, `7.x` -> +version 7) to select the correct Docker image for testing. +4. **Dump Generation:** Runs the regression test suite using the reusable +action with the `dump_db` option enabled, which creates a +`*_postgres_sqldump.tar` file. +5. **Artifact Upload:** Uploads the generated SQL dump archive as a named +artifact (e.g., `sqldump_ggdb7_ubuntu`) to the workflow run. + +This workflow ensures that a current database schema dump is available as an +artifact following successful CI runs on the primary branches `main` and `7.x`. + ## Configuration The workflow is parameterized to support flexibility: diff --git a/.github/workflows/greengage-abi-tests.yml b/.github/workflows/greengage-abi-tests.yml index 3a72e300a72e..58e8b6f67997 100644 --- a/.github/workflows/greengage-abi-tests.yml +++ b/.github/workflows/greengage-abi-tests.yml @@ -8,22 +8,7 @@ concurrency: on: workflow_dispatch: pull_request: - paths: - - 'concourse/scripts/**' - - 'src/**' - - '.github/workflows/**' - - '.github/scripts/**' - - '.abi-check/**' - - push: - branches: - - 6X_STABLE - paths: - - 'concourse/scripts/**' - - 'src/**' - - '.github/workflows/**' - - '.github/scripts/**' - - '.abi-check/**' + branches: ['**'] env: # workaround required for checkout@v3, https://github.com/actions/checkout/issues/1590 @@ -95,8 +80,9 @@ jobs: - name: Install dependencies and abi-dumper run: | sudo gpdb_src/README.ubuntu.bash - sudo apt install -y libuv1-dev libpam0g-dev libldap-dev libipc-run-perl abi-dumper + sudo apt install -y libuv1-dev libpam0g-dev libldap-dev libipc-run-perl abi-dumper python-pip sudo ln -fs python2 /usr/bin/python + sudo python -m pip install future - name: Build Greengage run: | diff --git a/.github/workflows/greengage-ci.yml b/.github/workflows/greengage-ci.yml index 7e5fbbc7c41b..b3d307d6c4b4 100644 --- a/.github/workflows/greengage-ci.yml +++ b/.github/workflows/greengage-ci.yml @@ -6,7 +6,7 @@ on: branches: ['main'] # Trigger on push to main (after merged PR) tags: ['6.*'] # Trigger on tags for versioned releases pull_request: - branches: ['*'] # Trigger on pull requests for all branches + branches: ['**'] # Trigger on pull requests for all branches # Concurrency control to cancel previous runs on new push to same PR/branch concurrency: @@ -18,12 +18,12 @@ jobs: strategy: fail-fast: true # Stop on any failure in the matrix matrix: - target_os: [ubuntu] #, centos] + target_os: [ubuntu] permissions: contents: read # Explicit for default behavior packages: write # Required for GHCR access actions: write # Required for artifact upload - uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-build.yml@v1 + uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-build.yml@v19 with: version: 6 target_os: ${{ matrix.target_os }} @@ -36,12 +36,12 @@ jobs: strategy: fail-fast: true matrix: - target_os: [ubuntu] #, centos] + target_os: [ubuntu] permissions: contents: read # Explicit for default behavior packages: read # Explicit for GHCR access clarity actions: write # Required for artifact upload - uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-behave.yml@v1 + uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-behave.yml@v24 with: version: 6 target_os: ${{ matrix.target_os }} @@ -54,12 +54,12 @@ jobs: strategy: fail-fast: true matrix: - target_os: [ubuntu] #, centos] + target_os: [ubuntu] permissions: contents: read # Explicit for default behavior packages: read # Explicit for GHCR access clarity actions: write # Required for artifact upload - uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-regression.yml@v1 + uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-regression.yml@v19 with: version: 6 target_os: ${{ matrix.target_os }} @@ -72,12 +72,12 @@ jobs: strategy: fail-fast: true matrix: - target_os: [ubuntu] #, centos] + target_os: [ubuntu] permissions: contents: read # Explicit for default behavior packages: read # Explicit for GHCR access clarity actions: write # Required for artifact upload - uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-orca.yml@v1 + uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-orca.yml@v19 with: version: 6 target_os: ${{ matrix.target_os }} @@ -90,12 +90,12 @@ jobs: strategy: fail-fast: true matrix: - target_os: [ubuntu] #, centos] + target_os: [ubuntu] permissions: contents: read # Explicit for default behavior packages: read # Explicit for GHCR access clarity actions: write # Required for artifact upload - uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-resgroup.yml@v1 + uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-tests-resgroup.yml@v23 with: version: 6 target_os: ${{ matrix.target_os }} @@ -108,12 +108,12 @@ jobs: strategy: fail-fast: true matrix: - target_os: [ubuntu] #, centos] + target_os: [ubuntu] permissions: contents: read # Explicit for default behavior packages: write # Required for GHCR access actions: write # Required for artifact upload - uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-upload.yml@v1 + uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-upload.yml@v21 with: version: 6 target_os: ${{ matrix.target_os }} @@ -121,3 +121,22 @@ jobs: ghcr_token: ${{ secrets.GITHUB_TOKEN }} DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }} DOCKERHUB_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }} + + # Rebuild prod-redy version without debug extensions and pack it to deb + package: + needs: build # For push (main or tags) + strategy: + fail-fast: false + matrix: + target_os: [ubuntu] + permissions: + contents: read # Explicit for default behavior + packages: write # Required for GHCR access + actions: write # Required for artifact upload + uses: greengagedb/greengage-ci/.github/workflows/greengage-reusable-package.yml@v10 + with: + version: 6 + target_os: ${{ matrix.target_os }} + test_docker: ubuntu:22.04 # Docker Image (e.g., ubuntu:22.04, ubuntu:noble) for deploy test. Skip if empty + secrets: + ghcr_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/greengage-release.yml b/.github/workflows/greengage-release.yml new file mode 100644 index 000000000000..083834935c8e --- /dev/null +++ b/.github/workflows/greengage-release.yml @@ -0,0 +1,27 @@ +# Release creation workflow +name: Greengage release + +on: + release: + types: [released] + +jobs: + upload-to-release: + strategy: + fail-fast: false + matrix: + include: + - version: 6 + extensions: deb ddeb + artifact_name: deb-packages + runs-on: ubuntu-latest + permissions: + contents: write + actions: read + steps: + - name: Upload packages to release + uses: greengagedb/greengage-ci/.github/actions/upload-pkgs-to-release@v10 + with: + version: ${{ matrix.version }} + extensions: ${{ matrix.extensions }} + artifact_name: ${{ matrix.artifact_name }} diff --git a/.github/workflows/greengage-sql-dump.yml b/.github/workflows/greengage-sql-dump.yml new file mode 100644 index 000000000000..9c069189d0dd --- /dev/null +++ b/.github/workflows/greengage-sql-dump.yml @@ -0,0 +1,83 @@ +# .github/workflows/greengage-sql-dump.yml +name: Greengage SQL Dump + +on: + workflow_run: + workflows: ["Greengage CI"] + types: [completed] + branches: + - main + - 7.x + +jobs: + create-sql-dump-regression: + runs-on: ubuntu-latest + timeout-minutes: 180 + if: | + github.event.workflow_run.event == 'push' && + github.event.workflow_run.conclusion == 'success' && + !startsWith(github.event.workflow_run.head_branch, 'refs/tags/') + strategy: + fail-fast: true + matrix: + target_os: [ubuntu] + permissions: + contents: read # Explicit for default behavior + packages: read # Explicit for GHCR access clarity + actions: write # Required for cache and artifact upload + steps: + - name: Display trigger information + env: + RUN_NAME: ${{ github.event.workflow_run.name }} + RUN_ID: ${{ github.event.workflow_run.id }} + RUN_HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }} + RUN_HEAD_SHA: ${{ github.event.workflow_run.head_sha }} + RUN_HEAD_COMMIT_MESSAGE: ${{ github.event.workflow_run.head_commit.message }} + RUN_HEAD_COMMIT_AUTHOR: ${{ github.event.workflow_run.head_commit.author.name }} + run: | + cat << EOF + ================== Workflow Trigger Information ================== + Triggered by: ${RUN_NAME} (#${RUN_ID}) + Target branch: ${RUN_HEAD_BRANCH} + Source commit: ${RUN_HEAD_SHA} + Commit message: ${RUN_HEAD_COMMIT_MESSAGE} + + Author: ${RUN_HEAD_COMMIT_AUTHOR} + ================================================================== + EOF + + - name: Maximize disk space + uses: greengagedb/greengage-ci/.github/actions/maximize-disk-space@v19 + + - name: Determine version from branch + id: version + env: + BRANCH: ${{ github.event.workflow_run.head_branch }} + run: | + case "$BRANCH" in + main) + echo "version=6" >> $GITHUB_OUTPUT + ;; + 7.x) + echo "version=7" >> $GITHUB_OUTPUT + ;; + *) + echo "Unsupported branch: $BRANCH" + exit 1 + ;; + esac + + - name: Create SQL Dump via regression tests + uses: greengagedb/greengage-ci/.github/actions/tests/regression@v16 + with: + image: ghcr.io/${{ github.repository }}/ggdb${{ steps.version.outputs.version }}_${{ matrix.target_os }}:${{ github.event.workflow_run.head_sha }} + optimizer: postgres + target_os: ${{ matrix.target_os }} + dump_db: "true" + + - name: Upload SQL Dump + uses: actions/upload-artifact@v4 + with: + name: sqldump_ggdb${{ steps.version.outputs.version }}_${{ matrix.target_os }} + path: /mnt/logs/${{ matrix.target_os }}_postgres_sqldump.tar + if-no-files-found: error diff --git a/.gitignore b/.gitignore index 8f979721c160..799206a138da 100644 --- a/.gitignore +++ b/.gitignore @@ -53,6 +53,10 @@ GTAGS lib*dll.def lib*.pc compile_commands.json +**/debian/* +!**/debian/co* +!**/debian/lint* +!**/debian/rules # Local excludes in root directory /GNUmakefile @@ -65,3 +69,4 @@ compile_commands.json /Debug/ /Release/ /CMakeLists.txt +/Package/ diff --git a/.gitmodules b/.gitmodules index 1a1787514813..66c100cc7e2e 100644 --- a/.gitmodules +++ b/.gitmodules @@ -4,3 +4,6 @@ [submodule "gpcontrib/gpcloud/test/googletest"] path = gpcontrib/gpcloud/test/googletest url = https://github.com/google/googletest.git +[submodule "gpMgmt/bin/pythonSrc/PyGreSQL-5.2.5"] + path = gpMgmt/bin/pythonSrc/PyGreSQL-5.2.5 + url = https://github.com/PyGreSQL/PyGreSQL.git diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d3b512ee0dc8..a557b7874723 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,69 +1,77 @@ # Contributing -Greengage is maintained by a core team of developers with commit rights to the [main Greengage repository](https://github.com/GreengageDB/greengage) on GitHub. At the same time, we are very eager to receive contributions from anybody in the wider Greengage community. This section covers all you need to know if you want to see your code or documentation changes be added to Greengage and appear in the future releases. +Greengage was launched and is maintained by a team of independent contributors to Greenplum. Currently, we own the [Greengage DB repository on GitHub](https://github.com/GreengageDB/greengage), appoint the members of the architectural committee and perform other work related to the organizational process in addition to software development. However, great open source projects cannot exist and last long without a strong community, **so community contributions to Greengage DB are very welcome**. In this section, you’ll find initial guidance on how to contribute to Greengage and links to additional resources that will help you get your code released as part of Greengage. ## Getting started -Greengage is developed on GitHub, and anybody wishing to contribute to it will have to [have a GitHub account](https://github.com/signup) and be familiar with [Git tools and workflow](https://wiki.postgresql.org/wiki/Working_with_Git). +To contribute to Greengage, you will need a [GitHub account](https://github.com/signup). If you haven’t used Git before, find a time to familiarize yourself with [Git tooling and workflow](https://wiki.postgresql.org/wiki/Working_with_Git) before you start. -Once you have your GitHub account, [fork](https://github.com/GreengageDB/greengage/fork) this repository so that you can have your private copy to start hacking on and to use as source of pull requests. +A private copy of the Greengage repository is required to introduce changes. To create one, [fork](https://github.com/GreengageDB/greengage/fork) our repository and work on it. Having changed something, you will be able to pull request, and someone from the architectural committee will review your contribution. To get more information on the review process, see the “Patch review” section of this document. -## Licensing of Greengage contributions +## Contributions licensing -If the contribution you're submitting is original work, you can assume that we will release it as part of an overall Greengage release available to the downstream consumers under the Apache License, Version 2.0. However, in addition to that, we may also decide to release it under a different license (such as PostgreSQL License to the upstream consumers that require it. A typical example here would be we upstreaming your contribution back to PostgreSQL community (which can be done either verbatim or your contribution being upstreamed as part of the larger changeset). +As the original author of the code, you can expect that your contribution will be released and licensed under Apache License, v. 2.0. Additionally, certain contributions valuable to the broader PostgreSQL community might be released under the PostgreSQL license. If your patch is beneficial for upstream PostgreSQL, we can offer it for review individually or include it with a set of changes. -If the contribution you're submitting is NOT original work you have to indicate the name of the license and also make sure that it is similar in terms to the Apache License 2.0. Apache Software Foundation maintains a list of these licenses under [Category A](https://www.apache.org/legal/resolved.html#category-a). In addition to that, you may be required to make proper attribution in the [NOTICE file](https://github.com/GreengageDB/greengage/blob/adb-6.x/NOTICE) file similar to [these examples](https://github.com/GreengageDB/greengage/blob/adb-6.x/NOTICE#L278). +If you are NOT the author of the code you are contributing to Greengage, please make sure you take proper licensing into account. Check the third-party license terms for similarity to the Apache License 2.0. Similar licenses are listed on the Apache Software Foundation website under [Category A](https://www.apache.org/legal/resolved.html#category-a). Note that some of these licenses require making proper attribution in the [NOTICE file](https://github.com/GreengageDB/greengage/blob/main/NOTICE) (see examples [here](https://github.com/GreengageDB/greengage/blob/main/NOTICE#L335)). -Finally, keep in mind that it is NEVER a good idea to remove licensing headers from the work that is not your original one. Even if you are using parts of the file that originally had a licensing header at the top you should err on the side of preserving it. As always, if you are not quite sure about the licensing implications of your contributions, feel free to reach out to us. +Do NOT remove licensing headers from any piece of work done by a third party. Even partial usage of someone else’s work may assume licensing implications. Please give the original author credit for their work by keeping the licensing headers ## Coding guidelines -Your chances of getting feedback and seeing your code merged into the project greatly depend on how granular your changes are. If you happen to have a bigger change in mind, we highly recommend creating an issue first and sharing your proposal with us before you spend a lot of time writing code. Even when your proposal gets validated by the community, we still recommend doing the actual work as a series of small, self-contained commits. This makes the reviewer's job much easier and increases the timeliness of feedback. +Before introducing a major change, it is always good to validate your idea with the architectural committee. [Create an issue on GitHub](https://github.com/GreengageDB/greengage/issues) and explain what’s on your mind before you spend hours writing code. We expect that while explaining your proposal to the committee, you’ll be specific about the approaches you are going to use and reasons behind using them. -When it comes to C and C++ parts of Greengage, we try to follow [PostgreSQL Coding Conventions](https://www.postgresql.org/docs/devel/source.html). In addition to that: +Submitting changes in small portions is the best strategy, even if you are working on a massive feature. Smaller patches can be reviewed within a week while large changesets require more time for being checked by the committee members. To get timely feedback and see your code merged into the project faster, stick to small, granular pull requests. This is also a way to show the reviewers that their job is valued and respected. - * For C and perl code, please run pgindent if necessary as specified in [README.gpdb](/src/tools/pgindent/README.gpdb). - * All Python code must pass [Pylint](https://www.pylint.org/). - * All Go code must be formatted according to [gofmt](https://golang.org/cmd/gofmt/). +To help you with the process of coding and describing your pull requests for reviewers, we have created a separate [Pull Request Submission Guidelines](https://greengagedb.org/en/blog/contributing.html) document. Please refer to it when in doubt and contact us if the document provides no answer. -We recommend using `git diff --color` when reviewing your changes so that you don't have any spurious whitespace issues in the code that you submit. +Here we mention just a few best practices we expect you to apply while contributing to Greengage DB. For detailed recommendations, please refer to Greengage DB’s [Pull Request Submission Guidelines](https://greengagedb.org/en/blog/contributing.html). -All new functionality that is contributed to Greengage should be covered by regression tests that are contributed alongside it. If you are uncertain on how to test or document your work, please raise the question in a PR and the developer community will do its best to help you. + - Follow [PostgreSQL Coding Conventions](https://www.postgresql.org/docs/devel/source.html) when writing C/C++ code for Greengage. + - Run **pgindent** for C and Perl code as per [README.gpdb](https://github.com/GreengageDB/greengage/blob/main/src/tools/pgindent/README.gpdb). + - Use [Pylint](https://www.pylint.org/) for all Python code. + - Format all Golang code in accordance with [gofmt](https://golang.org/cmd/gofmt/). -At the very minimum you should always be running `make installcheck-world` to make sure that you're not breaking anything. +Use git `diff --color` as you review your changes to avoid spurious whitespace issues in the submitted code. -## Changes applicable to upstream PostgreSQL +Regression tests are mandatory for every new feature that you contribute to Greengage. All tests covering new functionality should also be contributed to the project. Check [Pull Request Submission Guidelines](https://greengagedb.org/en/blog/contributing.html) to make sure that all tests are placed in the right folders within the project repository. If you need guidance related to testing or documenting your contributions, please explicitly include your questions in the pull request, and the architectural committee members will address them during patch review. -If the change you're working on touches functionality that is common between PostgreSQL and Greengage, you may be asked to forward-port it to PostgreSQL. This is not only so that we keep reducing the delta between the two projects, but also so that any change that is relevant to PostgreSQL can benefit from a much broader review of the upstream PostgreSQL community. In general, it is a good idea to keep both code bases handy so you can be sure whether your changes may need to be forward-ported. +At the very minimum you should always make sure that all local test runs are successful before submitting a pull request (PR) to the main repository. + +## PostgreSQL-related changes + +We prefer to get the changes related to the shared functionality of PostgreSQL and Greengage DB reviewed by the members of both communities. The larger Postgres community has more resources to help improve your patches, that’s why we may request submitting your PostgreSQL-related changes to Postgres upstream to leverage that power and reduce the delta between Greengage DB and PostgreSQL. If your contribution is likely to be forward-ported to PostgreSQL, please refer to PostgreSQL code base where appropriate. ## Patch submission -Once you are ready to share your work with the Greengage core team and the rest of the Greengage community, you should push all the commits to a branch in your own repository forked from our one and [send us a pull request](https://help.github.com/articles/about-pull-requests/). +We expect that in the repository you forked from the Greengage DB one you’ll create a branch **other than main** that will contain the changes you prepared to share with us and the rest of the community. Then [send us a pull request](https://help.github.com/articles/about-pull-requests/). -We require all pull requests to be submitted against the main branch (clearly stating if the change needs to be back-ported to STABLE branches). If the change is ONLY applicable to given STABLE branch, you may decide to submit your pull requests against an active STABLE release branch. +Do keep in mind that your patch review will slow down + - if you do not follow [Pull Request Submission Guidelines](https://greengagedb.org/en/blog/contributing.html) + - if tests are missing or copied to incorrect folders -Things which slow down patch approval - - missing to accompany tests (or reproducible steps at minimum) - - submitting the patch against STABLE branch where the fix also applies to main branch +## Patch review -## Validation checks and CI +All submitted patches are subject to review by the architectural committee members. The time required for review depends of the volume and complexity of the submitted patch: + - Up to 1 week for small/easy patches; + - Up to 4 weeks for patches of medium complexity + - Up to 8 weeks for patches of extra size or complexity. -Once you submit your pull request, you will immediately see a number of validation checks performed by our automated CI pipelines. If any of these checks fails, you will need to update your pull request to take care of the issue. Pull requests with failed validation checks are very unlikely to receive any further peer review from the community members. +Architectural committee reserves the right to decline patches without review if they introduce no valuable changes and/or contain garbage code. -If you cannot figure out why a certain validation check failed, feel free to ask us in the pull request. +Each contributed patch should get approvals from two architectural committee members before being merged into the project. Both pull request description and code will be thoroughly examined to ensure high quality and security. -## Patch review +The first reviewer might initiate a discussion proposing further improvements / changes to your pull request. You might also be asked to explain certain solutions and approaches applied as part of the proposed changeset. Discussions around pull requests along with any other communication in the Greengage DB project are covered by the Code of Conduct. Once the patch is accepted by the first reviewer, the second reviewer steps up to double-check your contribution and possibly provide a final commentary. -A submitted pull request with passing validation checks is assumed to be available for peer review. Peer review is the process that ensures that contributions to Greengage are of high quality and align well with the road map and community expectations. Every member of the Greengage community is encouraged to review pull requests and provide feedback. Since you don't have to be a core team member to be able to do that, we recommend following a stream of pull reviews to anybody who's interested in becoming a long-term contributor to Greengage. As [Linus would say](https://en.wikipedia.org/wiki/Linus's_Law) "given enough eyeballs, all bugs are shallow". +Note that the members of committee volunteer to review patches. Therefore, their availability may be limited, and reasonable delays are possible. In many cases, being proactive and asking for feedback may speed up the review process. -One outcome of the peer review could be a consensus that you need to modify your pull request in certain ways. GitHub allows you to push additional commits into a branch from which a pull request was sent. Those additional commits will be then visible to all of the reviewers. +After basic review (including feature relevancy, absence of malicious changes, etc.) and approve to run CI pipelines, the processes of patch review begins. -A peer review converges when it receives at least one +1 and no -1s votes from the participants. At that point you should expect one of the core team members to pull your changes into the project. +## Validation checks and CI -Greengage prides itself on being a collaborative, consensus-driven environment. We do not believe in vetoes and any -1 vote casted as part of the peer review has to have a detailed technical explanation of what's wrong with the change. +Your patch will undergo a series of validation checks from our automated CI pipeline. If any of them fails, you will need to change the patch you contributed so as to pass this check next time. -At any time during the patch review, you may experience delays based on the availability of reviewers and core team members. Please be patient. That being said, don't get discouraged either. If you're not getting expected feedback for a few days add a comment asking for updates on the pull request itself. +While the process is generally intuitive and enables you understand what exactly was wrong, do not hesitate to ask your reviewers for help if you don’t know why your approved patch was not merged. Use the pull request discussion to contact them. ## Direct commits to the repository -On occasion you will see core team members committing directly to the repository without going through the pull request workflow. This is reserved for small changes only and the rule of thumb we use is this: if the change touches any functionality that may result in a test failure, then it has to go through a pull request workflow. If, on the other hand, the change is in the non-functional part of the code base (such as fixing a typo inside of a comment block) core team members can decide to just commit to the repository directly. +Members of the architectural committee may sometimes commit to the repository directly, without submitting pull requests. Usually they do so to introduce minor changes (i.e. typo corrections), all major code contributions need to be submitted as pull requests and go through checks. diff --git a/NOTICE b/NOTICE index 1bae549c9f76..1267a1240e33 100644 --- a/NOTICE +++ b/NOTICE @@ -1417,7 +1417,7 @@ distribute verbatim copies of this document is granted. ================================================================= - For: Python, PyGreSQL, argparse, python-subprocess32 + For: Python, argparse, python-subprocess32 ---------------------- Python License --------------------------- ================================================================= @@ -1471,6 +1471,42 @@ Agreement. +================================================================= +---------------------- PyGreSQL --------------------------------- +================================================================= + +Written by D'Arcy J.M. Cain (darcy@PyGreSQL.org) + +Based heavily on code written by Pascal Andre (andre@chimay.via.ecp.fr) + +Copyright (c) 1995, Pascal Andre + +Further modifications copyright (c) 1997-2008 by D'Arcy J.M. Cain + +Further modifications copyright (c) 2009-2023 by the PyGreSQL Development Team + +PyGreSQL is released under the PostgreSQL License, a liberal Open Source +license, similar to the BSD or MIT licenses: + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose, without fee, and without a written agreement +is hereby granted, provided that the above copyright notice and this +paragraph and the following two paragraphs appear in all copies. In +this license the term "AUTHORS" refers to anyone who has contributed code +to PyGreSQL. + +IN NO EVENT SHALL THE AUTHORS BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, +SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, +ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF +AUTHORS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +THE AUTHORS SPECIFICALLY DISCLAIM ANY WARRANTIES, INCLUDING, BUT NOT LIMITED +TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE +AUTHORS HAVE NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, +ENHANCEMENTS, OR MODIFICATIONS. + + ================================================================= ---------------------- pexpect ---------------------------------- ================================================================= diff --git a/README.CentOS.bash b/README.CentOS.bash index cd0aacabc261..bee249e73f57 100755 --- a/README.CentOS.bash +++ b/README.CentOS.bash @@ -14,17 +14,23 @@ sudo yum install -y \ libcurl-devel \ libevent-devel \ libkadm5 \ - libyaml-devel \ + libtool \ + libuuid-devel \ + libuv-devel \ libxml2-devel \ + libxslt-devel \ + libyaml-devel \ libzstd-devel \ + libzstd-static \ + net-tools \ + openldap-devel \ + openssl \ openssl-devel \ + pam-devel \ + perl-Env \ perl-ExtUtils-Embed \ python-devel \ python-pip \ readline-devel \ xerces-c-devel \ zlib-devel - -sudo pip install conan -sudo pip install -r python-dependencies.txt -sudo pip install -r python-developer-dependencies.txt diff --git a/README.Rhel-Rocky.bash b/README.Rhel-Rocky.bash new file mode 100755 index 000000000000..866c646e08fb --- /dev/null +++ b/README.Rhel-Rocky.bash @@ -0,0 +1,45 @@ +#!/bin/bash + +sudo dnf -y update +sudo dnf -y install epel-release +sudo dnf -y install 'dnf-command(config-manager)' +sudo dnf config-manager --set-enabled devel +sudo dnf makecache + +sudo dnf -y install\ + apr-devel \ + bison \ + bzip2-devel \ + cmake3 \ + flex \ + gcc \ + gcc-c++ \ + iproute \ + krb5-devel \ + libcurl-devel \ + libevent-devel \ + libicu \ + libkadm5 \ + libtool \ + libuuid-devel \ + libuv-devel \ + libxml2-devel \ + libxslt-devel \ + libyaml-devel \ + net-tools \ + openldap-devel \ + openssl \ + openssl-devel \ + pam-devel \ + perl-Env \ + perl-ExtUtils-Embed \ + perl-IPC-Run \ + perl-JSON \ + perl-Test-Base \ + procps-ng \ + python2-devel \ + python2-pip \ + readline-devel \ + snappy-devel \ + xerces-c-devel \ + zlib-devel diff --git a/README.linux.md b/README.linux.md index fb5e754faa1c..78b3da1a9e28 100644 --- a/README.linux.md +++ b/README.linux.md @@ -1,74 +1,49 @@ -## For CentOS: - -- Install Dependencies +## For CentOS 7: +- Install dependencies using README.CentOS.bash script: ```bash ./README.CentOS.bash ``` + Note: CentOS 7 is EOL — configure `yum` to use a valid repo (e.g., `vault.centos.org`) before installing dependencies. -- If you want to link cmake3 to cmake, run: +## For RHEL/Rocky 8: +- Install dependencies using README.Rhel-Rocky.bash script: ```bash - sudo ln -sf /usr/bin/cmake3 /usr/local/bin/cmake + ./README.Rhel-Rocky.bash ``` -- Make sure that you add `/usr/local/lib` and `/usr/local/lib64` to -`/etc/ld.so.conf`, then run command `ldconfig`. - -- If you want to install and use gcc-6 by default, run: - +- Build and install zstd with static library, e.g.: ```bash - sudo yum install -y centos-release-scl - sudo yum install -y devtoolset-6-toolchain - echo 'source scl_source enable devtoolset-6' >> ~/.bashrc + cd /tmp + curl -LO https://github.com/facebook/zstd/releases/download/v1.4.4/zstd-1.4.4.tar.gz + tar -xf zstd-1.4.4.tar.gz + cd zstd-1.4.4 + make -j$(nproc) + sudo make install PREFIX=/usr/local ``` -## For RHEL: - -- Install Development Tools. - - For RHEL 8: Install `Development Tools`: - - ```bash - sudo yum group install -y "Development Tools" - ``` - - - For RHEL versions (< 8.0): Install `devtoolset-7`: - - ```bash - sudo yum-config-manager --enable rhui-REGION-rhel-server-rhscl - sudo yum install -y devtoolset-7-toolchain - ``` - -- Install dependencies using README.CentOS.bash script. - - For RHEL 8: Execute additional steps before running README.CentOS.bash script. - - Note: Make sure installation of `Development Tools` includes `git` and `make` else install these tools manually. - - ```bash - sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm - sed -i -e 's/python-devel /python2-devel /' -e 's/python-pip/python2-pip/' -e 's/sudo pip/sudo pip2/' README.CentOS.bash - sed -i '/xerces-c-devel/d' README.CentOS.bash - sudo ln -s /usr/bin/python2.7 /usr/bin/python - ``` - - - Install dependencies using README.CentOS.bash script. +- Create symbolic link to Python 2 in `/usr/bin`: - ```bash - ./README.CentOS.bash - ``` + ```bash + sudo ln -s python2 /usr/bin/python + ``` -## For Ubuntu (versions 20.04 or 22.04): +## For Ubuntu (versions 22.04 or 24.04): - Install dependencies using README.ubuntu.bash script: ```bash sudo ./README.ubuntu.bash ``` -- Create symbolic link to Python 2 in `/usr/bin`: +- For Ubuntu 22.04, create symbolic link to Python 2 in `/usr/bin`: ```bash sudo ln -s python2 /usr/bin/python ``` + Note: Supported Python versions: 2.7 or 3.9 to 3.12. The version is selected + by the `python` command. For Ubuntu 24.04, Python3 is already configured + in `README.ubuntu.bash`. For Ubuntu 22.04, we recommend using Python2. - Ensure that your system supports American English with an internationally compatible character encoding scheme. To do this, run: ```bash @@ -85,19 +60,7 @@ ## Common Platform Tasks: -1. Create gpadmin and setup ssh keys - - Either use: - - ```bash - # Requires gpdb clone to be named gpdb_src - gpdb_src/concourse/scripts/setup_gpadmin_user.bash - ``` - to create the gpadmin user and set up keys, - - OR - - manually create ssh keys so you can do ssh localhost without a password, e.g., +1. Setup SSH keys so you can run ssh localhost without a password, e.g., ```bash ssh-keygen @@ -108,7 +71,7 @@ 2. Verify that you can ssh to your machine name without a password ```bash - ssh # e.g., ssh briarwood + ssh `hostname` # e.g., ssh briarwood ``` 3. Set up your system configuration: @@ -150,7 +113,3 @@ su - $USER # Apply settings ``` -5. Make sure that you download yaml and psutil as submodules. To do this, use `git clone --recurse-submodules` when downloading the source code. If you want to update the submodules, run: - ```bash - git submodule update --init --recursive --force - ``` diff --git a/README.md b/README.md index 16eb0e7e2b0d..b378f2b36b51 100644 --- a/README.md +++ b/README.md @@ -37,56 +37,84 @@ Follow [these macOS steps](README.macOS.md) for getting your system ready for GP ### Installing dependencies (for Linux developers) Follow [appropriate linux steps](README.linux.md) for getting your system ready for GPDB +### Downloading submodules +Make sure that you download submodules. To do this, use `git clone --recurse-submodules` +when downloading the source code. If you want to update the submodules, run: +```bash +git submodule update --init --recursive --force +``` + ### Build the database +The recommended way to build the database is to use build system located in the gpAux +directory, which is also used for CI testing and building packages. + +To create optimized release build, use the following: + +``` +make GPROOT=~/build PARALLEL_MAKE_OPTS=-j8 dist -C gpAux +``` +where `GPROOT` defines the base installation directory. + +By default, the target installation directory is `greengage-db-devel`. +You can override this directory name by setting the `GPDIR` option. +For example, to install into `/usr/local/gpdb`: `make GPROOT=/usr/local GPDIR=gpdb dist -C gpAux`. + +To run regression tests, a debug build with debug extensions is required. It can be built using the following command: + +``` +make GPROOT=~/build PARALLEL_MAKE_OPTS=-j8 devel -C gpAux ``` -# Configure build environment to install at /usr/local/gpdb -./configure --with-perl --with-python --with-libxml --with-gssapi --prefix=/usr/local/gpdb -# Compile and install -make -j8 -make -j8 install +Load the greengage environment into your current shell: -# Bring in greengage environment into your running shell -source /usr/local/gpdb/greengage_path.sh +``` +source ~/build/greengage-db-devel/greengage_path.sh +``` -# Start demo cluster +Start demo cluster: + +``` make create-demo-cluster -# (gpdemo-env.sh contains __PGPORT__ and __MASTER_DATA_DIRECTORY__ values) +``` + +To use the demo cluster, source the environment variables from gpdemo-env.sh, which contains +__PGPORT__ and __MASTER_DATA_DIRECTORY__ values: + +``` source gpAux/gpdemo/gpdemo-env.sh ``` The directory, the TCP ports, the number of segments, and the existence of standbys for segments and coordinator for the demo cluster can be changed -on the fly. +when starting the demo cluster. Instead of `make create-demo-cluster`, consider: ``` DATADIRS=/tmp/gpdb-cluster PORT_BASE=5555 NUM_PRIMARY_MIRROR_PAIRS=1 WITH_MIRRORS=false make create-demo-cluster ``` -The TCP port for the regression test can be changed on the fly: - +If you want to clean all generated files: ``` -PGPORT=5555 make installcheck-world +make distclean ``` -To turn GPORCA off and use Postgres planner for query optimization: +## Running tests + +* By default, tests use the GPORCA optimizer: ``` -set optimizer=off; +make installcheck-world ``` -If you want to clean all generated files +* To turn GPORCA off and use the Postgres planner for query optimization: ``` -make distclean +PGOPTIONS='-c optimizer=off' make installcheck-world ``` -## Running tests - -* The default regression tests +* The TCP port for the regression test can be changed: ``` -make installcheck-world +PGPORT=5555 make installcheck-world ``` * The top-level target __installcheck-world__ will run all regression @@ -112,17 +140,15 @@ make installcheck-world ## Alternative Configurations +Internally, the configure script is used to adapt the build system to the characteristics of the host machine. For finer control over the components that are built, configuration options must be specified explicitly. The configuration options in effect are recorded in the file config.log and may be inspected with the following command: +``` +head config.log +``` + ### Building GPDB without GPORCA Currently, GPDB is built with GPORCA by default. If you want to build GPDB without GPORCA, configure requires `--disable-orca` flag to be set. -``` -# Clean environment -make distclean - -# Configure build environment to install at /usr/local/gpdb -./configure --disable-orca --with-perl --with-python --with-libxml --prefix=/usr/local/gpdb -``` ### Building GPDB with gpperfmon enabled @@ -134,11 +160,11 @@ See [more information about gpperfmon here](gpAux/gpperfmon/README.md) gpperfmon is dependent on several libraries like apr, apu, and libsigar -### Building GPDB with Python3 enabled +### Building GPDB with plpython3u enabled GPDB supports Python3 with plpython3u UDF -See [how to enable Python3](src/pl/plpython/README.md) for details. +See [how to enable plpython3u](src/pl/plpython/README.md) for details. ### Building GPDB client tools on Windows diff --git a/README.ubuntu.bash b/README.ubuntu.bash index b740a108eb8d..1f2f758b6976 100755 --- a/README.ubuntu.bash +++ b/README.ubuntu.bash @@ -1,16 +1,24 @@ #!/bin/bash - +# Some packages, for example KRB5, not installing properly without this option +export DEBIAN_FRONTEND=noninteractive apt-get update apt-get install -y \ bison \ + build-essential \ cmake \ curl \ + debhelper \ + devscripts \ + dh-python \ + fakeroot \ flex \ g++ \ gcc \ git \ iproute2 \ iputils-ping \ + krb5-admin-server \ + krb5-kdc \ libapr1-dev \ libaprutil1-dev \ libbz2-dev \ @@ -34,7 +42,21 @@ apt-get install -y \ openssh-client \ openssh-server \ pkg-config \ - python2 \ - python2-dev \ + protobuf-compiler \ + python3-dev \ rsync \ + sudo \ zlib1g-dev + +if [ "$(lsb_release -si)" == "Ubuntu" ] && [ "$(lsb_release -sr)" == "22.04" ]; then + apt-get install -y \ + python-pip \ + python2 \ + python2-dev + python2 -m pip install future==0.16 +else + apt-get install -y \ + python3-pip \ + python-is-python3; + python -m pip install future==1.0.0 +fi diff --git a/README.windows.md b/README.windows.md index c2f84e5f117d..d407a395ecfa 100644 --- a/README.windows.md +++ b/README.windows.md @@ -142,7 +142,7 @@ cmake --build . --config Release --target INSTALL 4. Build pygresql, needed by gpload ``` -cd \gpMgmt\bin\pythonSrc\PyGreSQL-4.0 +cd \gpMgmt\bin\pythonSrc\PyGreSQL-5.2.5 mkdir build cd build cmake -DCMAKE_PREFIX_PATH=C:\greengage-db-devel -DCMAKE_INSTALL_PREFIX:PATH=C:\greengage-db-devel -G "Visual Studio 15 2017 Win64" .. diff --git a/ci/Dockerfile b/ci/Dockerfile index 06c0213aa2ee..9667b890477d 100644 --- a/ci/Dockerfile +++ b/ci/Dockerfile @@ -36,7 +36,7 @@ RUN yum makecache && \ # The installation goes through a bootstrap script since pip from the # repository can no longer update itself. curl https://bootstrap.pypa.io/pip/2.7/get-pip.py | python && \ - pip install psi pytest==3.10.1 && \ + pip install psi pytest==3.10.1 mock==3.0.5 future==0.16 && \ pip install allure-behave==2.4.0 && \ yum clean all diff --git a/ci/Dockerfile.ubuntu b/ci/Dockerfile.ubuntu index eb7b658ac788..bc6824f86712 100644 --- a/ci/Dockerfile.ubuntu +++ b/ci/Dockerfile.ubuntu @@ -1,4 +1,10 @@ -FROM ubuntu:22.04 as base +ARG OS_VERSION=22.04 + +FROM ubuntu:${OS_VERSION} AS base +SHELL ["/bin/bash", "-o", "pipefail", "-c"] +WORKDIR /home/gpadmin + +ENV PIP_BREAK_SYSTEM_PACKAGES=1 ARG UBUNTU_MAIN_MIRROR=http://archive.ubuntu.com/ubuntu ARG UBUNTU_UPDATES_MIRROR=http://archive.ubuntu.com/ubuntu @@ -12,6 +18,7 @@ ARG adb_python3=https://downloads.adsw.io/ADB/6.27.1_arenadata56/ubuntu/22.04/co ARG adb_python3_bin=/opt/adb6-python3.9/bin/python3 COPY README.ubuntu.bash ./ +COPY gpMgmt/bin/gpload_test/pytest_requirement.txt ./ RUN set -eux; \ sed -i \ -e "s|http://archive.ubuntu.com/ubuntu/ jammy |${UBUNTU_MAIN_MIRROR}/ jammy |g" \ @@ -24,9 +31,10 @@ RUN set -eux; \ wget $sigar $sigar_headers $adb_python3; \ apt install -y ./*.deb; \ rm ./*.deb; \ - ln -s python2 /usr/bin/python; \ +# Set Python 2 as default when needed + test "$(lsb_release -sr)" == "22.04" && ln -s python2 /usr/bin/python; \ # Install pg_bsd_indent used by pgindent utility - wget https://ftp.postgresql.org/pub/dev/pg_bsd_indent-1.3.tar.gz -O - | tar -xzf -; \ + wget --progress=dot:giga --no-hsts https://ftp.postgresql.org/pub/dev/pg_bsd_indent-1.3.tar.gz -O - | tar -xzf -; \ make install -C pg_bsd_indent; \ rm -r pg_bsd_indent; \ # The en_US.UTF-8 locale is needed to run GPDB @@ -35,46 +43,45 @@ RUN set -eux; \ mkdir /run/sshd; \ # Alter precedence in favor of IPv4 during resolving echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf; \ -# Packages for tests - DEBIAN_FRONTEND=noninteractive \ - apt install -y krb5-kdc krb5-admin-server fakeroot sudo python-pip \ - openjdk-11-jdk protobuf-compiler; \ # Install allure-behave for behave tests - pip2 install allure-behave==2.4.0; \ - pip2 cache purge; \ + python -m pip install --no-cache-dir allure-behave==2.4.0 mock==3.0.5 && \ +# Install pytest for gpload test + python -m pip install --no-cache-dir -r pytest_requirement.txt && \ + rm pytest_requirement.txt && \ # ADCC extension dependencies - $adb_python3_bin -m pip install protobuf==3.20.0; \ - $adb_python3_bin -m pip cache purge + $adb_python3_bin -m pip install --no-cache-dir protobuf==3.20.0 && \ +# Cleanup to reduce image size + apt-get clean && \ + rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /var/cache/man/ -WORKDIR /home/gpadmin +ENV LANG=en_US.UTF-8 ENV CONFIGURE_FLAGS="--enable-debug-extensions --with-gssapi --enable-cassert --enable-debug --enable-depend" -FROM base as build +FROM base AS build COPY . gpdb_src -RUN mkdir /home/gpadmin/bin_gpdb +RUN mkdir bin_gpdb -ENV TARGET_OS=ubuntu -ENV OUTPUT_ARTIFACT_DIR=bin_gpdb -# Use python3.9 to compile into GPDB's plpython3u -ENV PYTHON3=$adb_python3_bin +ENV TARGET_OS=ubuntu \ + OUTPUT_ARTIFACT_DIR=bin_gpdb # Compile with running mocking tests -RUN bash /home/gpadmin/gpdb_src/concourse/scripts/compile_gpdb.bash +# Use python3 to compile into GPDB's plpython3u +RUN test "$(lsb_release -sr)" == "22.04" && export PYTHON3=$adb_python3_bin; \ + gpdb_src/concourse/scripts/compile_gpdb.bash -FROM base as code +FROM base AS code # Use --exclude, when it will be available in stable syntax. COPY . gpdb_src RUN rm -rf gpdb_src/.git/ -FROM base as test -COPY --from=code /home/gpadmin/gpdb_src gpdb_src -COPY --from=build /home/gpadmin/bin_gpdb /home/gpadmin/bin_gpdb +FROM base AS test + +COPY --from=code /home/gpadmin/gpdb_src gpdb_src +COPY --from=build /home/gpadmin/bin_gpdb bin_gpdb +COPY --from=build /home/gpadmin/gpdb_src/VERSION gpdb_src # Install entab used by pgindent utility. # This should be done using gpdb sources. RUN make -C gpdb_src/src/tools/entab install clean - -# Volume for tests output -VOLUME /home/gpadmin/gpdb_src/src/test diff --git a/ci/docker-compose.yaml b/ci/docker-compose.yaml index 550e24b28d24..5d98e5a567ec 100644 --- a/ci/docker-compose.yaml +++ b/ci/docker-compose.yaml @@ -1,64 +1,56 @@ ---- +x-common-service: &common-service + image: "${IMAGE}" + privileged: true + sysctls: + kernel.sem: 500 1024000 200 4096 + init: true + ulimits: + nofile: 65535 + entrypoint: > + sleep infinity -version: "3" services: cdw: - image: "${IMAGE}" + <<: *common-service working_dir: /home/gpadmin hostname: cdw volumes: - - "$PWD/ssh_keys/id_rsa:/home/gpadmin/.ssh/id_rsa" - - "$PWD/ssh_keys/id_rsa.pub:/home/gpadmin/.ssh/id_rsa.pub" - - "$PWD/allure-results:/tmp/allure-results" - - "$PWD/logs_cdw:/logs" - privileged: true - sysctls: - kernel.sem: 500 1024000 200 4096 - init: true - ulimits: - nofile: 65535 - entrypoint: > - sleep infinity + - ${PWD}/allure-results:/tmp/allure-results + - ${PWD}/ssh_keys:/home/gpadmin/.ssh.src + - ${PWD}/logs_cdw:/logs sdw1: - image: "${IMAGE}" - privileged: true + <<: *common-service hostname: sdw1 volumes: - - "$PWD/ssh_keys/id_rsa:/home/gpadmin/.ssh/id_rsa" - - "$PWD/ssh_keys/id_rsa.pub:/home/gpadmin/.ssh/id_rsa.pub" - - "$PWD/logs_sdw1:/logs" - sysctls: - kernel.sem: 500 1024000 200 4096 - init: true - ulimits: - nofile: 65535 - entrypoint: > - sleep infinity + - ${PWD}/ssh_keys:/home/gpadmin/.ssh.src + - ${PWD}/logs_sdw1:/logs sdw2: - image: "${IMAGE}" - privileged: true + <<: *common-service hostname: sdw2 volumes: - - "$PWD/ssh_keys/id_rsa:/home/gpadmin/.ssh/id_rsa" - - "$PWD/ssh_keys/id_rsa.pub:/home/gpadmin/.ssh/id_rsa.pub" - sysctls: - kernel.sem: 500 1024000 200 4096 - init: true - ulimits: - nofile: 65535 - entrypoint: > - sleep infinity + - ${PWD}/ssh_keys:/home/gpadmin/.ssh.src + - ${PWD}/logs_sdw2:/logs sdw3: - image: "${IMAGE}" - privileged: true + <<: *common-service hostname: sdw3 volumes: - - "$PWD/ssh_keys/id_rsa:/home/gpadmin/.ssh/id_rsa" - - "$PWD/ssh_keys/id_rsa.pub:/home/gpadmin/.ssh/id_rsa.pub" - sysctls: - kernel.sem: 500 1024000 200 4096 - init: true - ulimits: - nofile: 65535 - entrypoint: > - sleep infinity + - ${PWD}/ssh_keys:/home/gpadmin/.ssh.src + - ${PWD}/logs_sdw3:/logs + sdw4: + <<: *common-service + hostname: sdw4 + volumes: + - ${PWD}/ssh_keys:/home/gpadmin/.ssh.src + - ${PWD}/logs_sdw4:/logs + sdw5: + <<: *common-service + hostname: sdw5 + volumes: + - ${PWD}/ssh_keys:/home/gpadmin/.ssh.src + - ${PWD}/logs_sdw5:/logs + sdw6: + <<: *common-service + hostname: sdw6 + volumes: + - ${PWD}/ssh_keys:/home/gpadmin/.ssh.src + - ${PWD}/logs_sdw6:/logs diff --git a/ci/readme.md b/ci/readme.md index 070186b5a61b..788480991f0a 100644 --- a/ci/readme.md +++ b/ci/readme.md @@ -8,6 +8,12 @@ Change directory to gpdb sources destination. Make sure that directry doesn't co docker build -t gpdb6_regress:latest -f ci/Dockerfile.ubuntu . ``` +To build an image based on Ubuntu 24.04, specify the version in build args: + +```bash +docker build -t gpdb6_regress:latest --build-arg OS_VERSION=24.04 -f ci/Dockerfile.ubuntu . +``` + There are two additional options in [Dockerfile](./Dockerfile) to passthrough urls for [sigar](https://github.com/hyperic/sigar) packages: * `--build-arg sigar=https://path_to_sigar.rpm` for package with sigar library @@ -103,7 +109,9 @@ It required to add `gpMgmt/tests` directory to `PYTHONPATH`. Greengage cluster in Docker containers has its own peculiarities in preparing a cluster for tests. All tests are run in one way or another on the demo cluster, wherever possible. -For example, cross_subnet tests or tests with tag `concourse_cluster` currently not worked because of too complex cluster preconditions. +For example, cross_subnet tests currently not worked because of too complex cluster preconditions. +Behave tests run either on the concourse cluster (cluster on several hosts) or on the demo (cluster on single host), +while tests with both tags (concourse and demo) run only on the concourse cluster to avoid running the same test twice. Tests in a `docker compose` cluster use the same ssh keys for `gpadmin` user and pre-add the cluster hosts to `.ssh/know_hosts` and `/etc/hosts`. diff --git a/ci/scripts/behave_collect_logs.bash b/ci/scripts/behave_collect_logs.bash new file mode 100755 index 000000000000..16064acb744d --- /dev/null +++ b/ci/scripts/behave_collect_logs.bash @@ -0,0 +1,13 @@ +feature=$1 + +params=( + "./ d gpAdminLogs" + "gpdb_src/gpAux/gpdemo/datadirs/ d log" + "gpdb_src/gpAux/gpdemo/datadirs/ d pg_log" +) +for param in "${params[@]}"; do + read -r path type name <<< "$param" + [ -d "$path" ] && find "$path" -name "$name" -type "$type" \ + -exec tar -rf "/logs/behave_${feature}_${name}.tar" "{}" \; +done +chmod -R a+rwX /logs diff --git a/ci/scripts/behave_gpdb.bash b/ci/scripts/behave_gpdb.bash index 2d99d5444085..a8cbcd84c74f 100755 --- a/ci/scripts/behave_gpdb.bash +++ b/ci/scripts/behave_gpdb.bash @@ -11,8 +11,6 @@ function gen_env(){ source /usr/local/greengage-db-devel/greengage_path.sh - source gpdb_src/gpAux/gpdemo/gpdemo-env.sh - if [[ ${FEATURE} == "gpexpand" ]]; then mkdir -p /home/gpadmin/sqldump wget -nv https://downloads.adsw.io/misc/dump.sql.xz -O /home/gpadmin/sqldump/dump.sql.xz @@ -40,10 +38,6 @@ function _main() { exit 1 fi - # Run inside a subshell so it does not pollute the environment after - # sourcing greengage_path - time (make_cluster) - time gen_env time run_test diff --git a/ci/scripts/init_containers.sh b/ci/scripts/init_containers.sh index 3b30e3fd3fd5..a5c65faf7153 100644 --- a/ci/scripts/init_containers.sh +++ b/ci/scripts/init_containers.sh @@ -1,5 +1,5 @@ #!/bin/bash -set -eo pipefail +set -eox pipefail project="$1" @@ -18,6 +18,8 @@ for service in $services do docker compose -p $project -f ci/docker-compose.yaml exec -T \ $service bash -c "mkdir -p /data/gpdata && chmod -R 777 /data && + # each host should have its own copy of the (initially identical) files in .ssh + cp -rf .ssh.src .ssh && source gpdb_src/concourse/scripts/common.bash && install_gpdb && ./gpdb_src/concourse/scripts/setup_gpadmin_user.bash" & done @@ -30,3 +32,11 @@ do $service bash -c "ssh-keyscan ${services/$service/} >> /home/gpadmin/.ssh/known_hosts" & done wait + +# Add ip and host names of all cluster nodes to /etc/hosts +for service in $services +do + docker compose -p $project -f ci/docker-compose.yaml exec -T \ + $service bash -c "for HOST in $services; do echo \"\$(host \"\$HOST\" | grep 'has address' | head -n 1 | cut -d ' ' -f 4) \$HOST\" >>/etc/hosts; done" & +done +wait diff --git a/ci/scripts/run_behave_tests.bash b/ci/scripts/run_behave_tests.bash index cdcbdc474bff..cb73d097e8f9 100755 --- a/ci/scripts/run_behave_tests.bash +++ b/ci/scripts/run_behave_tests.bash @@ -3,15 +3,14 @@ set -x -o pipefail behave_tests_dir="gpMgmt/test/behave/mgmt_utils" -# TODO concourse_cluster tests are not stable -# clusters="concourse_cluster ~concourse_cluster,demo_cluster" +clusters="concourse_cluster ~concourse_cluster" -clusters="~concourse_cluster" +docker_compose_path="ci/docker-compose.yaml" if [ $# -eq 0 ] then # TODO cross_subnet and gpssh tests are excluded - features=`ls $behave_tests_dir -1 | grep feature | grep -v -E "cross_subnet|gpssh" | sed 's/\.feature$//'` + features=`ls $behave_tests_dir -1 | grep feature | sed 's/\.feature$//'` else for feature in $@ do @@ -45,18 +44,27 @@ run_feature() { echo "Started $feature behave tests on cluster $cluster and project $project" bash ci/scripts/init_containers.sh $project - docker compose -p $project -f ci/docker-compose.yaml exec -T \ + docker compose -p $project -f "$docker_compose_path" exec -T \ -e FEATURE="$feature" -e BEHAVE_FLAGS="--tags $feature --tags=$cluster \ -f behave_utils.ci.formatter:CustomFormatter \ -o non-existed-output \ -f allure_behave.formatter:AllureFormatter \ - -o /tmp/allure-results" \ + -o /tmp/allure-results \ + -f pretty" \ cdw gpdb_src/ci/scripts/behave_gpdb.bash status=$? - docker compose -p $project -f ci/docker-compose.yaml --env-file ci/.env down -v + if [ -n "$CI" ]; then + local services=$(docker compose -p $project -f "$docker_compose_path" config --services | tr '\n' ' ') + for service in $services; do + docker compose -p $project -f "$docker_compose_path" exec -T \ + $service /bin/bash -s "$feature" < ./ci/scripts/behave_collect_logs.bash + done + fi + + docker compose -p $project -f "$docker_compose_path" --env-file ci/.env down -v - if [[ $status > 0 ]]; then echo "Feature $feature failed with exit code $status"; fi + if [[ $status -gt 0 ]]; then echo "Feature $feature failed with exit code $status"; fi exit $status } diff --git a/ci/scripts/run_resgroup_test.bash b/ci/scripts/run_resgroup_test.bash index d7c96e8361ae..1cc66d9dddff 100755 --- a/ci/scripts/run_resgroup_test.bash +++ b/ci/scripts/run_resgroup_test.bash @@ -3,11 +3,16 @@ set -eox pipefail project="resgroup" +# Exit status file for cloud-init environments where exit codes aren't propagated. +# Parent processes can read this file to determine script success/failure. +logdir="$PWD/logs" +logfile=".exitcode" + function cleanup { docker compose -p $project -f ci/docker-compose.yaml --env-file ci/.env down } -mkdir ssh_keys -p +mkdir ssh_keys "$logdir" -p if [ ! -e "ssh_keys/id_rsa" ] then ssh-keygen -P "" -f ssh_keys/id_rsa @@ -50,7 +55,7 @@ docker compose -p $project -f ci/docker-compose.yaml exec -Tu gpadmin cdw bash - ${CONFIGURE_FLAGS} make -C /home/gpadmin/gpdb_src/src/test/regress - ssh sdw1 mkdir -p /home/gpadmin/gpdb_src/src/test/{regress,isolation2} "$logdir/$logfile" + docker compose -p $project -f ci/docker-compose.yaml exec -T cdw bash -ex < make GPROOT=`pwd` BLD_TARGETS="clients loaders connectivity gppkg" dist ## ---------------------------------------------------------------------- +## To build a Debian package: +## ---------------------------------------------------------------------- + + > make pkg-deb + +This will generate a changelog, build the Debian package using `debuild`, +and place the resulting `.deb` packages in the parent directory. + +See [README.package.md](README.package.md) + +## ---------------------------------------------------------------------- diff --git a/gpAux/Makefile b/gpAux/Makefile index f58c6bce3564..d2d9bf258e94 100644 --- a/gpAux/Makefile +++ b/gpAux/Makefile @@ -126,7 +126,9 @@ ORCA_CONFIG=--enable-orca rhel6_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs altlinux8.4_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs +altlinux10.2_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs astra1.7_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs +astra1.8_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs rhel7_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs rhel7_ppc64le_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs rhel8_x86_64_CONFIGFLAGS=--disable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs @@ -136,6 +138,7 @@ linux_x86_64_CONFIGFLAGS=${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --wit ubuntu18.04_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs ubuntu20.04_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs ubuntu22.04_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs +ubuntu24.04_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs sles12_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs redos7.3_x86_64_CONFIGFLAGS=--enable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy ${ORCA_CONFIG} --with-libxml --with-pythonsrc-ext --with-uuid=e2fs BLD_CONFIGFLAGS=$($(BLD_ARCH)_CONFIGFLAGS) @@ -170,34 +173,34 @@ RECONFIG : rm -f $(GPPGDIR)/GNUmakefile $(GPPGDIR)/GNUmakefile : $(GPPGDIR)/configure env.sh - rm -rf $(DESTDIR)$(INSTLOC) + rm -rf $(INSTLOC) mkdir -p $(GPPGDIR) echo "Running ./configure with CONFIGFLAGS=$(CONFIGFLAGS)\n" cd $(GPPGDIR) && CC="$(strip $(BLD_CC) $(BLD_CFLAGS))" \ CFLAGS=$(INSTCFLAGS) \ ./configure $(CONFIGFLAGS) \ - --prefix=$(INSTLOC) \ - --mandir=$(INSTLOC)/man + --prefix=$(DISTPATH) \ + --mandir=$(DISTPATH)/man Debug/GNUmakefile : $(GPPGDIR)/configure env.sh - rm -rf $(DESTDIR)$(INSTLOC) + rm -rf $(INSTLOC) mkdir -p Debug echo "Running ./configure with CONFIGFLAGS=$(CONFIGFLAGS)\n" cd Debug && CC="$(strip $(BLD_CC) $(BLD_CFLAGS))" \ CFLAGS=$(INSTCFLAGS) \ ./configure $(CONFIGFLAGS) \ - --prefix=$(INSTLOC) \ - --mandir=$(INSTLOC)/man + --prefix=$(DISTPATH) \ + --mandir=$(DISTPATH)/man Release/GNUmakefile : $(GPPGDIR)/configure env.sh - rm -rf $(DESTDIR)$(INSTLOC) + rm -rf $(INSTLOC) mkdir -p Release echo "Running ./configure with CONFIGFLAGS=$(CONFIGFLAGS)\n" cd Release && CC="$(strip $(BLD_CC) $(BLD_CFLAGS))" \ CFLAGS=$(INSTCFLAGS) \ ./configure $(CONFIGFLAGS) \ - --prefix=$(INSTLOC) \ - --mandir=$(INSTLOC)/man + --prefix=$(DISTPATH) \ + --mandir=$(DISTPATH)/man #--------------------------------------------------------------------- # autoconf @@ -224,8 +227,8 @@ perl_archlibexp:=$(shell perl -MConfig -e 'print $$Config{archlibexp}') # set default build steps define BUILD_STEPS - @rm -rf $(DESTDIR)$(INSTLOC) - cd $(BUILDDIR) && PYGRESQL_LDFLAGS='-Wl,-rpath,\$$$$ORIGIN/../../../lib -Wl,--enable-new-dtags' $(MAKE) $(PARALLEL_MAKE_OPTS) install + @rm -rf $(INSTLOC) + cd $(BUILDDIR) && PYGRESQL_LDFLAGS='-Wl,-rpath,\$$$$ORIGIN/../../lib -Wl,--enable-new-dtags' $(MAKE) $(PARALLEL_MAKE_OPTS) install cd $(BUILDDIR)/src/pl/plpython && $(MAKE) clean && $(MAKE) $(PARALLEL_MAKE_OPTS) install && cd $(BUILDDIR) cd $(BUILDDIR)/src/pl/plperl && $(MAKE) clean && echo "LDFLAGS += -Wl,-rpath,$(perl_archlibexp)/CORE -Wl,--enable-new-dtags" >> GNUmakefile && echo "LDFLAGS_SL += -Wl,-rpath,$(perl_archlibexp)/CORE -Wl,--enable-new-dtags" >> GNUmakefile && $(MAKE) $(PARALLEL_MAKE_OPTS) install && cd $(BUILDDIR) #@$(MAKE) mgmtcopy INSTLOC=$(INSTLOC) @@ -241,7 +244,7 @@ endef ifeq "$(BLD_GPDB_BUILDSET)" "partial" define BUILD_STEPS - rm -rf $(DESTDIR)$(INSTLOC) + rm -rf $(INSTLOC) cd $(BUILDDIR)/gpMgmt/ && $(MAKE) generate_greengage_path_file cd $(BUILDDIR)/src/backend/ && $(MAKE) ../../src/include/parser/gram.h cd $(BUILDDIR)/src/backend/ && $(MAKE) ../../src/include/utils/errcodes.h @@ -290,9 +293,9 @@ endif #--------------------------------------------------------------------- devel : INSTCFLAGS=$(DEBUGFLAGS) -devel : INSTLOC=$(DEVPATH) -devel : CLIENTSINSTLOC=$(CLIENTSDEVPATH) -devel : CONFIGFLAGS+= --enable-cassert --enable-debug --enable-depend +devel : INSTLOC=$(DESTDIR)$(DEVPATH) +devel : CLIENTSINSTLOC=$(DESTDIR)$(CLIENTSDEVPATH) +devel : CONFIGFLAGS+= --enable-cassert --enable-debug --enable-debug-extensions --enable-depend ifdef ENABLE_VPATH_BUILD devel : BUILDDIR=Debug devel : ISCONFIG=$(BUILDDIR)/GNUmakefile @@ -307,8 +310,8 @@ endif #--------------------------------------------------------------------- dist : INSTCFLAGS=$(OPTFLAGS) -dist : INSTLOC=$(DISTPATH) -dist : CLIENTSINSTLOC=$(CLIENTSDISTPATH) +dist : INSTLOC=$(DESTDIR)$(DISTPATH) +dist : CLIENTSINSTLOC=$(DESTDIR)$(CLIENTSDISTPATH) ifdef ENABLE_VPATH_BUILD dist : BUILDDIR=Release dist : ISCONFIG=$(BUILDDIR)/GNUmakefile @@ -319,8 +322,8 @@ endif $(BUILD_STEPS) dist_prof : INSTCFLAGS=$(PROFFLAGS) -dist_prof : INSTLOC=$(DISTPATH) -dist_prof : CLIENTSINSTLOC=$(CLIENTSDISTPATH) +dist_prof : INSTLOC=$(DESTDIR)$(DISTPATH) +dist_prof : CLIENTSINSTLOC=$(DESTDIR)$(CLIENTSDISTPATH) ifdef ENABLE_VPATH_BUILD dist_prof : BUILDDIR=Release dist_prof : ISCONFIG=$(BUILDDIR)/GNUmakefile @@ -331,7 +334,7 @@ dist_prof : gccVersionCheck GPROOTDEP RECONFIG $(ISCONFIG) endif $(BUILD_STEPS) -dist_faultinj : CLIENTSINSTLOC=$(CLIENTSDISTPATH) +dist_faultinj : CLIENTSINSTLOC=$(DESTDIR)$(CLIENTSDISTPATH) ifdef ENABLE_VPATH_BUILD dist_faultinj : BUILDDIR=Release dist_faultinj : ISCONFIG=$(BUILDDIR)/GNUmakefile @@ -342,6 +345,67 @@ dist_faultinj : gccVersionCheck GPROOTDEP RECONFIG $(ISCONFIG) endif $(BUILD_STEPS) +#--------------------------------------------------------------------- +# Packaging targets with changelog options +#--------------------------------------------------------------------- + +# Metadata vars +PACKAGE_NAME := $(shell grep '^Package:' debian/control | head -1 | awk '{print $$2}') +MAINTAINER := $(shell grep '^Maintainer:' debian/control | sed 's/Maintainer: //') +DATE_RFC := $(shell date -R) +ARTIFACTS_DIR := $(CURDIR)/../Package + +../VERSION : + @echo "Update $@" + ../getversion > $@ + @cat $@ + +version-vars : ../VERSION + $(eval FULL_VERSION := $(shell [ -f ../VERSION ] && perl -pe 's, ,-,g' ../VERSION)) + $(eval PACKAGE_VERSION := $(shell [ -f ../VERSION ] && perl -pe 's, .*,,g' ../VERSION)) + $(eval IS_RELEASE := $(if $(findstring +dev,$(PACKAGE_VERSION)),no,yes)) + $(eval STABILITY := $(if $(filter yes,$(IS_RELEASE)),stable,unstable)) + $(eval BUILD_TYPE := $(if $(filter yes,$(IS_RELEASE)),Release build,Development build)) + +version-info : version-vars + @echo "PACKAGE_VERSION: $(PACKAGE_VERSION)" + @echo "FULL_VERSION: $(FULL_VERSION)" + @echo "IS_RELEASE: $(IS_RELEASE)" + @echo "STABILITY: $(STABILITY)" + @echo "BUILD_TYPE: $(BUILD_TYPE)" + +# Generate package control files +changelog : debian/changelog +debian/changelog : version-vars + @echo "$(PACKAGE_NAME) ($(PACKAGE_VERSION)) $(STABILITY); urgency=low" > $@ + @echo "" >> $@ + @echo " * $(BUILD_TYPE)" >> $@ + @echo "" >> $@ + @echo " -- $(MAINTAINER) $(DATE_RFC)" >> $@ + +debian/install: + @echo "$(PACKAGE_NAME)/* /" > $@ + + +# Default packaging target +pkg : pkg-deb + +# Build Debian package +pkg-deb : GPROOT = /opt/greengagedb +pkg-deb : GPDIR = $(PACKAGE_NAME) +pkg-deb : debian/changelog debian/install + @echo "Building with GPROOT=$(GPROOT), GPDIR=$(GPDIR), PACKAGE_NAME=$(PACKAGE_NAME)" + @GPROOT="$(GPROOT)" GPDIR="$(GPDIR)" PACKAGE_NAME="$(PACKAGE_NAME)" debuild --preserve-env -us -uc -b + @mkdir -p $(ARTIFACTS_DIR) + @find $(CURDIR)/../ -maxdepth 1 -type f \( -name "*.deb" \ + -o -name "*.ddeb" \ + -o -name "*.build" \ + -o -name "*.buildinfo" \ + -o -name "*.changes" \) \ + -exec mv -f {} $(ARTIFACTS_DIR)/ \; + +.PHONY: pkg pkg-deb changelog version-vars version-info + #--------------------------------------------------------------------- # clientTools #--------------------------------------------------------------------- @@ -412,7 +476,7 @@ endif endif ifeq "$(INSTLOC)" "" -INSTLOC=$(GPDIR) +INSTLOC=$(DESTDIR)$(GPDIR) endif VERSION:=$(shell [ -f ../VERSION ] && perl -pe 's, ,-,g' ../VERSION) @@ -473,7 +537,7 @@ PVK_FILESET = \ $(NULL) pvk : INSTCFLAGS=$(OPTFLAGS) -pvk : INSTLOC=$(DISTPATH) +pvk : INSTLOC=$(DESTDIR)$(DISTPATH) pvk : gccVersionCheck version GPROOTDEP cd $(GPMGMT)/bin && $(MAKE) stream mkdir -p $(INSTLOC)/bin/lib @@ -539,7 +603,7 @@ ifeq "$(shell uname -s)" "Darwin" echo "pgbouncer can't build on Mac" else @if [ ! -f extensions/pgbouncer/source/configure ]; then cd extensions/pgbouncer/source && ./autogen.sh ;fi - @cd extensions/pgbouncer/source && ./configure --with-libevent=$(BLD_TOP)/ext/$(BLD_ARCH) --prefix=$(INSTLOC) --enable-evdns --with-pam --with-ldap + @cd extensions/pgbouncer/source && ./configure --with-libevent=$(BLD_TOP)/ext/$(BLD_ARCH) --prefix=$(DISTPATH) --enable-evdns --with-pam --with-ldap $(MAKE) -C extensions/pgbouncer/source install endif @@ -618,8 +682,8 @@ copylibs : mkdir -p $(INSTLOC)/include greengage_path: - mkdir -p $(DESTDIR)$(INSTLOC) - $(BUILDDIR)/gpMgmt/bin/generate-greengage-path.sh no > $(DESTDIR)$(INSTLOC)/greengage_path.sh + mkdir -p $(INSTLOC) + $(BUILDDIR)/gpMgmt/bin/generate-greengage-path.sh no > $(INSTLOC)/greengage_path.sh copylicense: for proddir in $(INSTLOC) $(CLIENTSINSTLOC); do \ @@ -669,7 +733,7 @@ distclean : @if [ -d Release -a -f Release/GNUmakefile ]; then cd Release; $(MAKE) distclean; fi @if [ -d $(GPPGDIR) -a -f $(GPPGDIR)/GNUmakefile ]; then cd $(GPPGDIR); $(MAKE) distclean; fi @if [ -d $(GPMGMT)/bin ]; then cd $(GPMGMT)/bin; $(MAKE) distclean; fi - @rm -f VERSION + @if [ -d .git ] ; then rm -f VERSION; fi @rm -f env.sh distprep : $(ISCONFIG) diff --git a/gpAux/Makefile.global b/gpAux/Makefile.global index 0adfa1c8c40f..88c3560d5199 100644 --- a/gpAux/Makefile.global +++ b/gpAux/Makefile.global @@ -48,8 +48,11 @@ rhel7_x86_64_BLD_CFLAGS=-m64 rhel8_x86_64_BLD_CFLAGS=-m64 rhel7_ppc64le_BLD_CFLAGS=-m64 -fasynchronous-unwind-tables -fsigned-char altlinux8.4_x86_64_BLD_CFLAGS=-m64 +altlinux10.2_x86_64_BLD_CFLAGS=-m64 astra1.7_x86_64_BLD_CFLAGS=-m64 +astra1.8_x86_64_BLD_CFLAGS=-m64 ubuntu22.04_x86_64_BLD_CFLAGS=-m64 +ubuntu24.04_x86_64_BLD_CFLAGS=-m64 redos7.3_x86_64_BLD_CFLAGS=-m64 BLD_CFLAGS=$($(BLD_ARCH)_BLD_CFLAGS) diff --git a/gpAux/README.package.md b/gpAux/README.package.md new file mode 100644 index 000000000000..14e82f666ec8 --- /dev/null +++ b/gpAux/README.package.md @@ -0,0 +1,131 @@ +# Greengage Database Packaging System + +## Overview + +This documentation describes the Debian packaging system for Greengage Database located in the `gpAux/` subdirectory. The system builds Debian packages using a custom Makefile and `debian/rules` file. + +## Location and Structure + +The packaging system is located in: + +```text +./gpAux/ +``` + +The main components are: + +- `Makefile` - Defines packaging targets, version management, and artifact collection +- `debian/rules` - Debian build rules with custom overrides +- `debian/control` - Package metadata and dependencies +- Other standard Debian packaging files + +## Key Components + +### Makefile Targets + +1. **Version Management**: + - `../VERSION`: Generates version file using `../getversion` + - `version-vars`: Sets build variables (`FULL_VERSION`, `PACKAGE_VERSION`, `IS_RELEASE`, `STABILITY`, `BUILD_TYPE`) from `../VERSION` + - `version-info`: Displays version information for debugging + +2. **Packaging Targets**: + - `pkg`: Default target (aliases to `pkg-deb`) + - `pkg-deb`: Builds Debian package, preserves environment variables, and collects specific artifacts (`.deb`, `.ddeb`, `.build`, `.buildinfo`, `.changes`) + - `changelog`: Generates `debian/changelog` using version variables + - `debian/install`: Creates installation manifest + +### Debian Rules File + +The `debian/rules` file uses debhelper (dh) with custom overrides: + +1. **Distribution-specific Dependencies**: + - Detects Ubuntu 22.04 and adds `python2.7` dependency + +2. **Build Process Overrides**: + - Skips standard configure and build steps + - Uses the project's `make dist` target for installation + - Unsets standard compiler flags to avoid conflicts + - Enables parallel builds using all available CPU cores + +3. **Control File Generation**: + - Adds Python dependencies for Ubuntu 22.04 + +## Usage + +### Building the Package + +From the project root directory, run: + +```bash +make -C ./gpAux pkg-deb +``` + +### Custom Installation Paths + +To customize installation paths, set environment variables: + +```bash +make -C ./gpAux pkg-deb GGROOT=/custom/path GPDIR=custom_dir +``` + +### Environment Variables + +- `GGROOT`: Installation root directory (default: `/opt/greengagedb`) +- `GPDIR`: Subdirectory under `GGROOT` (default: `` from `debian/control`) +- `ARTIFACTS_DIR`: Directory for build artifacts (default: `$(CURDIR)/../Package`) + +## Build Process Details + +1. **Version Generation**: + - Runs `../getversion` to create `../VERSION` + - Processes version string into `FULL_VERSION` and `PACKAGE_VERSION` + - Sets `IS_RELEASE` and `STABILITY` for changelog generation + +2. **Package Building**: + - Executes `debuild` with preserved environment variables (`GGROOT`, `GPDIR`, `PACKAGE_NAME`) + - Skips signing with `-us -uc` flags + - Collects specific build artifacts (`.deb`, `.ddeb`, `.build`, `.buildinfo`, `.changes`) into `ARTIFACTS_DIR` + +3. **Installation**: + - Uses `make dist` for installation into `debian/tmp` + - Generates file manifest in `debian/install` + +## Dependencies + +The packaging system automatically handles: + +- `Ubuntu 22.04` detection and `python2.7` dependency +- Other distributions may require manual dependency configuration + +## Maintenance + +### Updating Package Metadata + +Edit `debian/control` to update: + +- Package description +- Maintainer information +- General dependencies + +### Adding Distribution Support + +Modify distribution detection in `debian/rules`: + +```makefile +ifeq ($(LSB_SI),Ubuntu) + ifeq ($(LSB_SR),22.04) + DEPS=python2.7 + endif + ifeq ($(LSB_SR),20.04) + DEPS=python2.7 + endif +endif +``` + +## Notes + +- Skips tests (`DEB_BUILD_OPTIONS=nocheck`) for faster builds +- Unsets compiler flags to avoid conflicts with the project's build system +- Enables parallel builds using all available CPU cores +- Builds without signing for development convenience +- Collects only specific build artifacts (`.deb`, `.ddeb`, `.build`, `.buildinfo`, `.changes`) into `$(CURDIR)/../Package` diff --git a/gpAux/client/scripts/greengage_clients_path.sh b/gpAux/client/scripts/greengage_clients_path.sh index 4715acbbf98c..f7356c164295 100644 --- a/gpAux/client/scripts/greengage_clients_path.sh +++ b/gpAux/client/scripts/greengage_clients_path.sh @@ -21,6 +21,6 @@ fi if [ "$1" != "-q" ]; then type python >/dev/null 2>&1 if [ $? -ne 0 ]; then - echo "Warning: Python not found. Python-2.5.1 or better is required to run gpload." + echo "Warning: Python not found. Python 2.7 or later is required to run gpload." fi fi diff --git a/gpAux/debian/compat b/gpAux/debian/compat new file mode 100644 index 000000000000..b1bd38b62a08 --- /dev/null +++ b/gpAux/debian/compat @@ -0,0 +1 @@ +13 diff --git a/gpAux/debian/control b/gpAux/debian/control new file mode 100644 index 000000000000..ff2de8895713 --- /dev/null +++ b/gpAux/debian/control @@ -0,0 +1,28 @@ +Source: greengage +Maintainer: Greengage +Section: database +Build-Depends: debhelper (>= 13), + dh-python, + lsb-release + +Package: greengage +Architecture: any +Depends: ${shlibs:Depends}, + iproute2, + iputils-ping, + less, + openssh-client, + openssh-server, + openssl, + rsync, + zip, + net-tools, + ${pythonRequires} +Conflicts: greengage-loaders, ${pythonConflicts} +Description: Greengage MPP database engine + Greengage Database (GPDB) is an advanced, fully featured, open + source data warehouse, based on PostgreSQL. It provides powerful and + rapid analytics on petabyte scale data volumes. Uniquely geared toward + big data analytics, Greengage Database is powered by the world’s most + advanced cost-based query optimizer delivering high analytical query + performance on large data volumes. diff --git a/gpAux/debian/copyright b/gpAux/debian/copyright new file mode 100644 index 000000000000..cdc1800d9a0a --- /dev/null +++ b/gpAux/debian/copyright @@ -0,0 +1,24 @@ +Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: greengage +Upstream-Contact: GreengageDB Team +Source: https://github.com/GreengageDB/greengage + +Files: * +Copyright: See COPYRIGHT file +License: Apache-2.0 + +License: Apache-2.0 + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + . + On Debian systems, the complete text of the Apache License 2.0 can be + found in "/usr/share/common-licenses/Apache-2.0". diff --git a/gpAux/debian/lintian-overrides b/gpAux/debian/lintian-overrides new file mode 100644 index 000000000000..491c4105a411 --- /dev/null +++ b/gpAux/debian/lintian-overrides @@ -0,0 +1,9 @@ +# 3rd-party Package +dir-or-file-in-opt + +# TODO +unusual-interpreter python +script-uses-unversioned-python-in-shebang + +# Not applicable for plugins +shared-library-lacks-prerequisites diff --git a/gpAux/debian/rules b/gpAux/debian/rules new file mode 100755 index 000000000000..7efdeb668de5 --- /dev/null +++ b/gpAux/debian/rules @@ -0,0 +1,46 @@ +#!/usr/bin/make -f +# Run command: `make -C ./gpdb_src/gpAux pkg-deb` + +DH_VERBOSE = 1 +export DH_OPTIONS = -v +export DEB_BUILD_OPTIONS = nocheck + +LSB_SI := $(shell lsb_release -si) +LSB_SR := $(shell lsb_release -sr) + +ifeq ($(LSB_SI),Ubuntu) + ifeq ($(LSB_SR),22.04) + DEPS = python2,python2.7 + CONFLICTS = python-is-python3 + else + DEPS = python3 + endif +else + DEPS = python2,python2.7 + CONFLICTS = python-is-python3 +endif + +%: + dh $@ + +override_dh_auto_clean: + @echo "Skipping clean" + +override_dh_auto_configure: + @echo "Skipping configure" + +override_dh_auto_build: + @echo "Skipping autobuild" + +override_dh_auto_install: + @echo "=== Building and installing with GPROOT=$(GPROOT), GPDIR=$(GPDIR), PACKAGE_NAME=$(PACKAGE_NAME) ===" + env -u CFLAGS -u CPPFLAGS -u CXXFLAGS -u LDFLAGS \ + $(MAKE) dist \ + DESTDIR=$(CURDIR)/debian/tmp/$(PACKAGE_NAME) \ + GPROOT=$(GPROOT) \ + GPDIR=$(GPDIR) \ + PARALLEL_MAKE_OPTS=-j$(shell nproc) + +override_dh_gencontrol: + @echo "=== GENCONTROL GPROOT=$(GPROOT), GPDIR=$(GPDIR), PACKAGE_NAME=$(PACKAGE_NAME) ===" + dh_gencontrol -p$(PACKAGE_NAME) -- -VpythonRequires="$(DEPS)" -VpythonConflicts="$(CONFLICTS)" diff --git a/gpAux/gpdemo/demo_cluster.sh b/gpAux/gpdemo/demo_cluster.sh index 26715322123f..68740c1408fb 100755 --- a/gpAux/gpdemo/demo_cluster.sh +++ b/gpAux/gpdemo/demo_cluster.sh @@ -392,11 +392,13 @@ if [ -n "${STATEMENT_MEM}" ]; then EOF fi -if [ -n "${OPTIMIZER_ENABLE_TABLE_ALIAS}" ]; then - cat >> $CLUSTER_CONFIG_POSTGRES_ADDONS<<-EOF - optimizer_enable_table_alias = ${OPTIMIZER_ENABLE_TABLE_ALIAS} - EOF -fi +# Demo cluster always works with disabled optimizer_enable_table_alias, so it's +# easy to run tests suite with no additional setup. +# TODO: remove this and GUC optimizer_enable_table_alias when tests are +# corrected for aliases. +cat >> $CLUSTER_CONFIG_POSTGRES_ADDONS<<-EOF +optimizer_enable_table_alias = false +EOF if [ "${ONLY_PREPARE_CLUSTER_ENV}" == "true" ]; then echo "ONLY_PREPARE_CLUSTER_ENV set, generated clusterConf file: $CLUSTER_CONFIG, exiting" diff --git a/gpAux/gpdemo/gpsegwalrep.py b/gpAux/gpdemo/gpsegwalrep.py index 490f66b739f9..412c8e3aae02 100755 --- a/gpAux/gpdemo/gpsegwalrep.py +++ b/gpAux/gpdemo/gpsegwalrep.py @@ -24,7 +24,9 @@ Assuming all of the above, you can just run the tool as so: ./gpsegwalrep.py [init|start|stop|destroy] """ +from __future__ import print_function +from builtins import object import argparse import os import sys @@ -49,29 +51,29 @@ def runcommands(commands, thread_name, command_finish, exit_on_error=True): output.append('%s: Running command... %s' % (datetime.datetime.now(), command)) with THREAD_LOCK: output = output + subprocess.check_output(command, stderr=subprocess.STDOUT, shell=True).split('\n') - except subprocess.CalledProcessError, e: + except subprocess.CalledProcessError as e: output.append(str(e)) output.append(e.output) if exit_on_error: with PRINT_LOCK: for line in output: - print '%s: %s' % (thread_name, line) - print '' + print('%s: %s' % (thread_name, line)) + print('') sys.exit(e.returncode) output.append('%s: %s' % (datetime.datetime.now(), command_finish)) with PRINT_LOCK: for line in output: - print '%s: %s' % (thread_name, line) - print '' + print('%s: %s' % (thread_name, line)) + print('') def displaySegmentConfiguration(): commands = [] commands.append("psql postgres -c \"select * from gp_segment_configuration order by content, dbid\"") runcommands(commands, "", "") -class InitMirrors(): +class InitMirrors(object): ''' Initialize the WAL replication mirror segment ''' def __init__(self, cluster_config, hostname, init=True): @@ -131,7 +133,7 @@ def run(self): for thread in initThreads: thread.join() -class StartInstances(): +class StartInstances(object): ''' Start a greengage segment ''' def __init__(self, cluster_config, host, operation, wait=False): @@ -211,7 +213,7 @@ def run(self): for thread in startThreads: thread.join() -class StopInstances(): +class StopInstances(object): ''' Stop all segments''' def __init__(self, cluster_config): @@ -246,7 +248,7 @@ def run(self): for thread in stopThreads: thread.join() -class DestroyMirrors(): +class DestroyMirrors(object): ''' Destroy the WAL replication mirror segment ''' def __init__(self, cluster_config): @@ -295,7 +297,7 @@ def run(self): for thread in destroyThreads: thread.join() -class GpSegmentConfiguration(): +class GpSegmentConfiguration(object): ROLE_PRIMARY = 'p' ROLE_MIRROR = 'm' STATUS_DOWN = 'd' @@ -314,7 +316,7 @@ def __init__(self, dbid, content, port, datadir, role, preferred_role, status, m self.status = status self.mode = mode -class ClusterConfiguration(): +class ClusterConfiguration(object): ''' Cluster configuration ''' def __init__(self, hostname, port, dbname, role = "all", status = "all", include_master = True, content = "all"): @@ -364,15 +366,15 @@ def refresh(self): query = ("SELECT dbid, content, port, datadir, role, preferred_role, status, mode " "FROM gp_segment_configuration s WHERE 1 = 1") - print '%s: fetching cluster configuration' % (datetime.datetime.now()) + print('%s: fetching cluster configuration' % (datetime.datetime.now())) dburl = dbconn.DbURL(self.hostname, self.port, self.dbname) - print '%s: fetched cluster configuration' % (datetime.datetime.now()) + print('%s: fetched cluster configuration' % (datetime.datetime.now())) try: with dbconn.connect(dburl, utility=True, unsetSearchPath=False) as conn: resultsets = dbconn.execSQL(conn, query).fetchall() - except Exception, e: - print e + except Exception as e: + print(e) sys.exit(1) contentIDs = Set() @@ -406,7 +408,7 @@ def refresh(self): contentIDs.add(seg_config.content) self.num_contents = len(contentIDs) - print 'found %d distinct content IDs' % (self.num_contents) + print('found %d distinct content IDs' % (self.num_contents)) def check_status_and_mode(self, expected_status, expected_mode): ''' Check if all the instance reached the expected_state and expected_mode ''' @@ -470,9 +472,9 @@ def WaitForRecover(cluster_configuration, max_retries = 200): number_of_segments = len(cluster_configuration.seg_configs) - print "cmd_all_sync: %s" % cmd_all_sync - print "cmd_find_error: %s" % cmd_find_error - print "number of contents: %s " % number_of_segments + print("cmd_all_sync: %s" % cmd_all_sync) + print("cmd_find_error: %s" % cmd_find_error) + print("number of contents: %s " % number_of_segments) retry_count = 1 while (retry_count < max_retries): @@ -487,7 +489,7 @@ def WaitForRecover(cluster_configuration, max_retries = 200): else: retry_count += 1 - print "WARNING: Incremental recovery took longer than expected!" + print("WARNING: Incremental recovery took longer than expected!") cmd_find_recovering = ("psql postgres -A -R ',' -t -c \"SELECT gp_segment_id" " FROM gp_stat_replication" " WHERE gp_segment_id in (%s) and sync_error = 'none'\"" % @@ -586,7 +588,7 @@ def ForceFTSProbeScan(cluster_configuration, expected_status = None, expected_mo StopInstances(cluster_config).run() elif args.operation == 'rebuild': if args.content is None: - print "ERROR: missing argument 'content' for rebuild operation" + print("ERROR: missing argument 'content' for rebuild operation") sys.exit(1) cluster_config_mirror = ClusterConfiguration(args.host, args.port, args.database, content=args.content, role=GpSegmentConfiguration.ROLE_MIRROR) diff --git a/gpAux/gpperfmon/src/gpmon/gpmon_catqrynow.py b/gpAux/gpperfmon/src/gpmon/gpmon_catqrynow.py index dc6f73da684a..890dd18a48a2 100644 --- a/gpAux/gpperfmon/src/gpmon/gpmon_catqrynow.py +++ b/gpAux/gpperfmon/src/gpmon/gpmon_catqrynow.py @@ -1,3 +1,4 @@ +from __future__ import print_function import os, sys, time GPMONDIR = 'gpperfmon/data' @@ -49,4 +50,4 @@ line[-3] = '"' + appname + '"' line[-2] = '"' + rsqname + '"' line[-1] = '"' + priority + '"' - print '|'.join(line).strip() + print('|'.join(line).strip()) diff --git a/gpAux/gpperfmon/src/gpmon/gpperfmon.sql b/gpAux/gpperfmon/src/gpmon/gpperfmon.sql index 0886aa77ac92..d80bee2ced84 100644 --- a/gpAux/gpperfmon/src/gpmon/gpperfmon.sql +++ b/gpAux/gpperfmon/src/gpmon/gpperfmon.sql @@ -148,7 +148,7 @@ create external web table public._database_tail ( create external web table public.master_data_dir (hostname text, dir text) -execute E'python -c "import socket, os; print socket.gethostname() + \\"|\\" + os.getcwd()"' on master +execute E'python -c "import socket, os; print(socket.gethostname() + \\"|\\" + os.getcwd())"' on master format 'csv' (delimiter '|'); diff --git a/gpAux/gpperfmon/src/gpmon/gpperfmon_install b/gpAux/gpperfmon/src/gpmon/gpperfmon_install index 0ae407345f06..ca57ac73fb05 100755 --- a/gpAux/gpperfmon/src/gpmon/gpperfmon_install +++ b/gpAux/gpperfmon/src/gpmon/gpperfmon_install @@ -18,7 +18,9 @@ USAGE: gpperfmon_install --port GPDB_PORT [--enable --password GPMON_PASSWORD] --gpperfmonport sets the guc 'gpperfmon_port' for gpperfmon communication (default is 8888) --verbose will show output from sub-commands ''' +from __future__ import print_function +from builtins import object import os, sys, time, re from subprocess import Popen @@ -28,12 +30,12 @@ try: from gppylib.userinput import ask_input from gppylib.gplog import get_default_logger, setup_tool_logging from gppylib.commands.unix import getLocalHostname, getUserName -except ImportError, e: +except ImportError as e: sys.exit('Cannot import modules. Please check that you have sourced greengage_path.sh. Detail: ' + str(e)) EXECNAME = os.path.split(__file__)[-1] -class Command: +class Command(object): def __init__(self, cmdstr, showOutput=False): self.cmd = cmdstr self.verbose = showOutput @@ -46,7 +48,7 @@ def run_command(cmd, verbose=False): cmdstr = "%s >& /dev/null" % cmdstr # hide password if bool(re.search('CREATE ROLE.*ENCRYPTED PASSWORD',cmdstr)): - regex = re.compile('ENCRYPTED\sPASSWORD\s\'(.*)\'') + regex = re.compile(r"ENCRYPTED\sPASSWORD\s'(.*)'") logger.info(regex.sub('ENCRYPTED PASSWORD \'********\'',cmdstr)) elif bool(re.search('echo.*:gpperfmon:gpmon:',cmdstr)): regex = re.compile(':gpperfmon:gpmon:(.*)\"') @@ -79,7 +81,7 @@ def cli_help(): def usage(): - print cli_help() or __doc__ + print(cli_help() or __doc__) ###### main() diff --git a/gpAux/releng/set_bld_arch.sh b/gpAux/releng/set_bld_arch.sh index efe799020d8d..4eab100dd9fc 100755 --- a/gpAux/releng/set_bld_arch.sh +++ b/gpAux/releng/set_bld_arch.sh @@ -7,11 +7,17 @@ case "`uname -s`" in if [ -f /etc/redhat-release -a ! -f /etc/altlinux-release -a ! -f /etc/redos-release ]; then case "`cat /etc/redhat-release`" in *) - BLD_ARCH_HOST="rhel`cat /etc/redhat-release | sed -e 's/CentOS Linux/RedHat/' -e 's/Red Hat Enterprise Linux/RedHat/' -e 's/WS//' -e 's/Server//' -e 's/Client//' | awk '{print $3}' | awk -F. '{print $1}'`_`uname -m | sed -e s/i686/x86_32/`" + BLD_ARCH_HOST="rhel`cat /etc/redhat-release | sed -e 's/CentOS Linux/RedHat/' -e 's/Red Hat Enterprise Linux/RedHat/' -e 's/Rocky Linux/RedHat/' -e 's/WS//' -e 's/Server//' -e 's/Client//' | awk '{print $3}' | awk -F. '{print $1}'`_`uname -m | sed -e s/i686/x86_32/`" ;; esac fi - + if [ -f /etc/altlinux-release ]; then + case "`cat /etc/altlinux-release`" in + *) + BLD_ARCH_HOST="$(. /etc/os-release; echo ${ID}${VERSION_ID} | sed 's/-/_/' | cut -d'.' -f1,2)_$(uname -m)" + ;; + esac + fi if [ -f /etc/astra_version ]; then BLD_ARCH_HOST="$(. /etc/os-release; echo ${ID}${VERSION_ID} | sed 's/-/_/')" fi diff --git a/gpMgmt/Makefile b/gpMgmt/Makefile index 4d0bfc397e2b..f098ff9ec522 100644 --- a/gpMgmt/Makefile +++ b/gpMgmt/Makefile @@ -22,8 +22,13 @@ install: generate_greengage_path_file if [ -e bin/ext/psutil ]; then \ cp -rp bin/ext/psutil $(DESTDIR)$(prefix)/lib/python ; \ fi + # Move _pg extension to sys.path root for Python 3 import (from _pg import *) if [ -e bin/ext/pygresql ]; then \ cp -rp bin/ext/pygresql $(DESTDIR)$(prefix)/lib/python ; \ + sofile=`find "$(DESTDIR)$(prefix)/lib/python/pygresql" -maxdepth 1 -type f -name "_pg*.so" | head -n 1`; \ + if [ -n "$$sofile" ]; then \ + mv -f "$$sofile" "$(DESTDIR)$(prefix)/lib/python/"; \ + fi; \ fi if [ -e bin/ext/yaml ]; then \ cp -rp bin/ext/yaml $(DESTDIR)$(prefix)/lib/python ; \ diff --git a/gpMgmt/Makefile.behave b/gpMgmt/Makefile.behave index b9b7c2feed1e..9295e0e13e11 100644 --- a/gpMgmt/Makefile.behave +++ b/gpMgmt/Makefile.behave @@ -1,5 +1,6 @@ PEXPECT_LIB=$(GPHOME)/bin/lib TEST_DIR=$(CURDIR)/test +STEPS_DIR=$(TEST_DIR)/behave/mgmt_utils/steps # To pass in custom flags as behave args(such as multiple flags), # use flags=--tags=foo, --tags=-bar @@ -7,9 +8,9 @@ behave: @which behave || (echo "behave not found" && exit 1) @echo "Running behave on management scripts..." @if [ -n """$(flags)""" ]; then \ - PYTHONPATH=$$PYTHONPATH:$(PEXPECT_LIB):$(TEST_DIR) behave $(CURDIR)/test/behave/* --no-source --no-skipped $(flags) 2>&1 ; \ + PYTHONPATH=$$PYTHONPATH:$(PEXPECT_LIB):$(TEST_DIR):$(STEPS_DIR) behave $(CURDIR)/test/behave/* --no-source --no-skipped $(flags) 2>&1 ; \ elif [ -n """$(tags)""" ]; then \ - PYTHONPATH=$$PYTHONPATH:$(PEXPECT_LIB):$(TEST_DIR) behave $(CURDIR)/test/behave/* --no-source --no-skipped --tags=$(tags) 2>&1 ; \ + PYTHONPATH=$$PYTHONPATH:$(PEXPECT_LIB):$(TEST_DIR):$(STEPS_DIR) behave $(CURDIR)/test/behave/* --no-source --no-skipped --tags=$(tags) 2>&1 ; \ else \ echo "Please specify tags=tagname or flags=[behave flags]"; \ exit 1; \ diff --git a/gpMgmt/bin/Makefile b/gpMgmt/bin/Makefile index 565941916ddc..a291c94cee43 100644 --- a/gpMgmt/bin/Makefile +++ b/gpMgmt/bin/Makefile @@ -71,7 +71,7 @@ endif # # PyGreSQL # -PYGRESQL_VERSION=4.0 +PYGRESQL_VERSION=5.2.5 PYGRESQL_DIR=PyGreSQL-$(PYGRESQL_VERSION) pygresql: @@ -80,7 +80,7 @@ pygresql: unset LIBPATH; \ ./generate-greengage-path.sh > $(DESTDIR)$(prefix)/greengage_path.sh ; \ fi - PATH=$(bindir):$$PATH && \ + PATH=$(DESTDIR)/$(bindir):$$PATH && \ if [ `uname -s` = 'OpenBSD' ]; then \ cd $(PYLIB_SRC)/$(PYGRESQL_DIR) && DESTDIR="$(DESTDIR)" CC=cc python setup.py build; \ else \ @@ -143,8 +143,6 @@ LOGILAB_ASTNG_DIR=logilab-astng-$(LOGILAB_ASTNG_VERSION) LOGILAB_COMMON_VERSION=0.50.1 LOGILAB_COMMON_DIR=logilab-common-$(LOGILAB_COMMON_VERSION) PYLINT_PYTHONPATH=$(PYLIB_DIR):$(PYLIB_SRC_EXT)/$(PYLINT_DIR)/build/lib/ -MOCK_VERSION=1.0.1 -MOCK_DIR=mock-$(MOCK_VERSION) SETUP_TOOLS_VERSION=36.6.0 PARSE_VERSION=1.8.2 ARG_PARSE_VERSION=1.2.1 @@ -155,8 +153,6 @@ PYTHONSRC_INSTALL=$(PYLIB_SRC_EXT)/install PYTHON_VERSION=$(shell python -c "import sys; print ('%s.%s' % (sys.version_info[0:2]))") PYTHONSRC_INSTALL_SITE=$(PYLIB_SRC_EXT)/install/lib/python$(PYTHON_VERSION)/site-packages PYTHONSRC_INSTALL_PYTHON_PATH=$(PYTHONPATH):$(PYTHONSRC_INSTALL_SITE) -# TODO: mock-1.0.1-py2.6.egg package should be updated. -MOCK_BIN=$(PYTHONSRC_INSTALL)/lib/python$(PYTHON_VERSION)/site-packages/mock-1.0.1-py2.6.egg UBUNTU_PLATFORM=$(shell if lsb_release -a 2>/dev/null | grep -q 'Ubuntu' ; then echo "Ubuntu"; fi) pylint: @@ -172,18 +168,6 @@ pylint: @touch $(PYLIB_SRC_EXT)/$(PYLINT_DIR)/build/lib/__init__.py @touch $(PYLIB_SRC_EXT)/$(PYLINT_DIR)/build/lib/logilab/__init__.py -$(MOCK_BIN): - @echo "--- mock for platform $(UBUNTU_PLATFORM)" - @if [ $(UBUNTU_PLATFORM) = "Ubuntu" ]; then\ - wget https://bootstrap.pypa.io/pip/2.7/get-pip.py && \ - python2 get-pip.py && \ - pip install mock==1.0.1;\ - else\ - mkdir -p $(PYTHONSRC_INSTALL_SITE) && \ - cd $(PYLIB_SRC_EXT)/ && tar xzf $(MOCK_DIR).tar.gz && \ - cd $(PYLIB_SRC_EXT)/$(MOCK_DIR)/ && \ - PYTHONPATH=$(PYTHONSRC_INSTALL_PYTHON_PATH) python setup.py install --prefix $(PYTHONSRC_INSTALL) ; \ - fi; PYTHON_FILES=`grep -l --exclude=Makefile --exclude=gplogfilter "/bin/env python" *`\ `grep -l "/bin/env python" $(SRC)/../sbin/*`\ @@ -201,16 +185,16 @@ check-regress: @PYTHONPATH=$(SRC):$(SRC)/ext:$(PYTHONPATH) \ gppylib/gpunit discover --verbose -s gppylib -p "test_regress*.py" 2> $(SRC)/../gpMgmt_testregress_results.log 1> $(SRC)/../gpMgmt_testregress_output.log -check: $(MOCK_BIN) +check: @echo "Running pure unit and also "unit" tests that require cluster to be up..." - @TMPDIR=/tmp PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR):$(PYLIB_DIR)/mock-1.0.1 \ - gppylib/gpunit discover --verbose -s $(SRC)/gppylib -p "test_unit*.py" 2> $(SRC)/../gpMgmt_testunit_results.log 1> $(SRC)/../gpMgmt_testunit_output.log - @TMPDIR=/tmp PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR):$(PYLIB_DIR)/mock-1.0.1 \ - gppylib/gpunit discover --verbose -s $(SRC)/gppylib -p "test_cluster*.py" 2>> $(SRC)/../gpMgmt_testunit_results.log 1>> $(SRC)/../gpMgmt_testunit_output.log + @TMPDIR=/tmp PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR) \ + gppylib/gpunit discover --verbose -s $(SRC)/gppylib -p "test_unit*.py" 2> $(SRC)/../gpMgmt_testunit_results.log 1> $(SRC)/../gpMgmt_testunit_output.log || (cat $(SRC)/../gpMgmt_testunit_results.log && cat $(SRC)/../gpMgmt_testunit_output.log && exit 1) + @TMPDIR=/tmp PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR) \ + gppylib/gpunit discover --verbose -s $(SRC)/gppylib -p "test_cluster*.py" 2>> $(SRC)/../gpMgmt_testunit_results.log 1>> $(SRC)/../gpMgmt_testunit_output.log || (cat $(SRC)/../gpMgmt_testunit_results.log && cat $(SRC)/../gpMgmt_testunit_output.log && exit 1) unitdevel: @echo "Running pure unit tests..." - PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR):$(PYLIB_DIR)/mock-1.0.1 \ + PYTHONPATH=$(SERVER_SRC):$(SERVER_SBIN):$(PYTHONPATH):$(PYTHONSRC_INSTALL_PYTHON_PATH):$(SRC)/ext:$(SBIN_DIR):$(LIB_DIR) \ python -m unittest discover --verbose -s $(SRC)/gppylib -p "test_unit*.py" @@ -220,7 +204,7 @@ installcheck-bash: ./test/suite.bash .PHONY: installcheck -installcheck: installcheck-bash +installcheck: check installcheck-bash $(MAKE) -C gpload_test $@ clean distclean: diff --git a/gpMgmt/bin/README b/gpMgmt/bin/README index ea0ff98334b1..206ccefa7321 100644 --- a/gpMgmt/bin/README +++ b/gpMgmt/bin/README @@ -17,9 +17,16 @@ should do the trick. Python Version -------------- -* The current utilities bundle python 2.5.1. We are planning on moving - forward to 2.6.1 in the near future. +* Supported Python versions: 2.7 or 3.9 to 3.12. +* To switch the Python version used during the build, change the `python` + command to point to the desired version. For example: + +``` +sudo ln -sf python3 /usr/bin/python +``` + +After changing the Python version, clean the build directory and recompile. Where Things Go --------------- @@ -155,10 +162,9 @@ Testing Management Scripts This directory contains the unit tests for the management scripts. These tests require the following Python modules to be installed: mock and pygresql. -These modules can be installed by running "git submodule update --init --recursive" -if they are not already installed on your machine. +These modules can be installed by running "make install" in gpMgmt/bin. -If you installed the dependencies using the above git command, you can run the tests with +If you installed the dependencies using the command above, you can run the tests with make, using the following commands in the current directory: "make check" will run all of the unit tests, some of which require a GPDB cluster to diff --git a/gpMgmt/bin/analyzedb b/gpMgmt/bin/analyzedb index 9b8977e2bf48..f47d3a6dd027 100755 --- a/gpMgmt/bin/analyzedb +++ b/gpMgmt/bin/analyzedb @@ -11,9 +11,13 @@ # import mainUtils FIRST to get python version check +from __future__ import division from gppylib.mainUtils import * +from builtins import map +from builtins import range +from past.utils import old_div from optparse import OptionParser -from Queue import Queue, Empty +from queue import Queue, Empty import os import re import shutil @@ -34,7 +38,7 @@ try: from gppylib.operations.unix import CheckDir, CheckFile, MakeDir from pygresql import pg -except ImportError, e: +except ImportError as e: sys.exit('Cannot import modules. Please check that you have sourced greengage_path.sh. Detail: ' + str(e)) EXECNAME = 'analyzedb' @@ -175,9 +179,11 @@ def validate_schema_exists(pg_port, dbname, schema): def execute_sql(query, master_port, dbname): dburl = dbconn.DbURL(port=master_port, dbname=dbname) - conn = dbconn.connect(dburl) - cursor = dbconn.execSQL(conn, query) - return cursor.fetchall() + + with dbconn.connect(dburl) as conn: + cursor = dbconn.execSQL(conn, query) + result = cursor.fetchall() + return result def get_lines_from_file(fname): content = [] @@ -235,8 +241,8 @@ def get_partition_state_tuples(pg_port, dbname, catalog_schema, partition_info): logger.info("Table %s.%s (%s) no longer exists and will not be analyzed", schemaname, partition_name, tupletable) else: logger.error(str(e)) - # If there's an exception, the transaction is closed so we need to reconnect - conn = dbconn.connect(dburl) + # If there's an exception, the transaction is aborted so we need to rollback it + conn.rollback() else: num_sqls += 1 if num_sqls == 1000: # The choice of batch size was chosen arbitrarily @@ -261,9 +267,9 @@ class AnalyzeDb(Operation): def __init__(self, options, args): if args: - logger.warn("Please note that some of the arguments (%s) aren't valid and will be ignored.", args) + logger.warning("Please note that some of the arguments (%s) aren't valid and will be ignored.", args) if not options.rootstats: - logger.warn("The --skip_root_stats option is no longer supported and will be ignored.") + logger.warning("The --skip_root_stats option is no longer supported and will be ignored.") if options.masterDataDirectory is None: options.masterDataDirectory = gp.get_masterdatadir() self.master_datadir = options.masterDataDirectory @@ -463,7 +469,7 @@ class AnalyzeDb(Operation): self._write_report(curr_ao_state, curr_last_op, heap_partitions, input_col_dict, root_partition_col_dict, dirty_partitions, target_list) logger.info("Done.") - except Exception, ex: + except Exception as ex: logger.exception(ex) raise @@ -593,7 +599,7 @@ class AnalyzeDb(Operation): tablenames = parse_tables_from_file(self.conn, self.config_file) canonical_tables = validate_tables(self.conn, tablenames) all_root_partitions = run_sql(self.conn, GET_ALL_ROOT_PARTITION_TABLES_SQL) - cfg_file = open(self.config_file, 'rU') + cfg_file = open(self.config_file, OPEN_READ_MODE) for line in cfg_file: # XXX: The file format does not allow listing tables with spaces in the name, # even when quoted @@ -610,7 +616,7 @@ class AnalyzeDb(Operation): for schema_table in alltables: col_dict[(schema_table[0], schema_table[1])] = set(['-1']) - return col_dict.keys() + return list(col_dict.keys()) def _get_include_or_exclude_cols(self, line_tokens, option_str): """ @@ -697,7 +703,7 @@ class AnalyzeDb(Operation): qresult = run_sql(self.conn, GET_REQUESTED_LAST_OP_INFO_SQL % oid_str) ret = [] for r in qresult: - tup = (r[0], r[1], str(r[2]), r[3], r[4], r[5]) + tup = (r[0], r[1], str(r[2]), r[3], r[4], str(r[5])) ret.append(tup) return ret @@ -748,7 +754,7 @@ class AnalyzeDb(Operation): self.dbname, current_time) logger.info("Writing last operation file %s" % last_operation_filename) - lines_to_write = map((lambda x: '%s,%s,%s,%s,%s,%s' % (x[0], x[1], x[2], x[3], x[4], x[5])), last_op_output) + lines_to_write = list(map((lambda x: '%s,%s,%s,%s,%s,%s' % (x[0], x[1], x[2], x[3], x[4], x[5])), last_op_output)) write_lines_to_file(last_operation_filename, lines_to_write) logger.debug("Verifying last operation file ...") verify_lines_in_file(last_operation_filename, lines_to_write) @@ -884,7 +890,7 @@ class AnalyzeDb(Operation): 2. The leaf partitions (if range partitioned, especially by date) will be ordered in descending order of the partition key, so that newer partitions can be analyzed first. """ - candidate_regclass_str = get_oid_str(itertools.chain(candidates, root_partition_col_dict.keys())) + candidate_regclass_str = get_oid_str(itertools.chain(candidates, list(root_partition_col_dict.keys()))) qresult = run_sql(self.conn, ORDER_CANDIDATES_BY_OID_SQL % candidate_regclass_str) ordered_candidates = [] for schema_tbl in qresult: @@ -966,7 +972,7 @@ def run_sql(conn, query): try: cursor = dbconn.execSQL(conn, query) res = cursor.fetchall() - except Exception, db_err: + except Exception as db_err: raise ExceptionNoStackTraceNeeded("%s" % db_err.__str__()) # .split('\n')[0]) cursor.close() return res @@ -1026,14 +1032,14 @@ def get_latest_analyze_timestamp(master_datadir, statefile_dir, dbname): files = sorted(os.listdir(analyze_dir)) if len(files) == 0: - logger.warn('Analyze state file directory %s is empty. Ignoring this directory...' % analyze_dir) + logger.warning('Analyze state file directory %s is empty. Ignoring this directory...' % analyze_dir) continue analyze_report_files = fnmatch.filter(files, 'analyze_[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_report') if len(analyze_report_files) == 0: - logger.warn( + logger.warning( 'No analyze report files found in analyze directory %s. Ignoring this directory...' % analyze_dir) continue for report_file in analyze_report_files: @@ -1051,7 +1057,7 @@ def get_prev_ao_state(timestamp, master_datadir, analyze_dir, dbname): # Parse the lines into (schemaname, tablename, modcount) tuples. Each line is # a comma-separated string. XXX: This file format cannot deal with names # with commas. - return map((lambda x: x.split(',')), lines) + return list(map((lambda x: x.split(',')), lines)) def get_prev_last_op(timestamp, master_datadir, analyze_dir, dbname): @@ -1082,7 +1088,7 @@ def get_prev_col_state(timestamp, master_datadir, analyze_dir, dbname): prev_col_dict = {} for line in lines: toks = line.strip().split(',') - toks = map(str.strip, toks) + toks = list(map(lambda s: s.strip(), toks)) prev_col_dict[(toks[0], toks[1])] = set(toks[2:]) return prev_col_dict @@ -1112,7 +1118,7 @@ def create_last_op_dict(last_op_entries): def construct_entries_from_dict_aostate(ao_state_dict): ret = [] - for key, item in ao_state_dict.iteritems(): + for key, item in ao_state_dict.items(): schema = key[0] table = key[1] ret.append("%s,%s,%s" % (schema, table, item)) @@ -1121,15 +1127,15 @@ def construct_entries_from_dict_aostate(ao_state_dict): def construct_entries_from_dict_lastop(last_op_dict): ret = [] - for value in last_op_dict.itervalues(): - for entry in value.itervalues(): + for value in last_op_dict.values(): + for entry in value.values(): ret.append(entry) return ret def construct_entries_from_dict_colstate(prev_col_dict): ret = [] - for schema_table, col_set in prev_col_dict.iteritems(): + for schema_table, col_set in prev_col_dict.items(): schema = schema_table[0] table = schema_table[1] cols = ','.join(col_set) @@ -1200,7 +1206,7 @@ def validate_dir(path): else: try: MakeDir(path).run() - except OSError, e: + except OSError as e: logger.exception("Could not create directory %s" % path) raise AnalyzeDirCreateFailed() else: @@ -1208,14 +1214,14 @@ def validate_dir(path): try: with tempfile.TemporaryFile(dir=path) as f: pass - except Exception, e: + except Exception as e: logger.exception("Cannot write to %s" % path) raise AnalyzeDirNotWritable() # Parse a list of tables from the config file. def parse_tables_from_file(conn, include_file): - in_file = open(include_file, 'rU') + in_file = open(include_file, OPEN_READ_MODE) line_no = 1 tables = [] for line in in_file: @@ -1250,7 +1256,7 @@ def validate_tables(conn, tablenames): # we validate the tables in batches of 1500 # XXX: We use a VALUES list now. What's the maximum size of that? batch_size = 1500 - nbatches = (len(tablenames) - 1) / batch_size + 1 + nbatches = old_div((len(tablenames) - 1), batch_size) + 1 curr_batch = 0 canonical_dict = {} @@ -1430,7 +1436,7 @@ class AnalyzeWorker(Worker): self.pool.addFinishedWorkItem(self.cmd) self.cmd = None - except Exception, e: + except Exception as e: self.logger.exception(e) if self.cmd: self.logger.debug("[%s] finished cmd with exception: %s" % (self.name, self.cmd)) diff --git a/gpMgmt/bin/el8_migrate_locale/el8_migrate_locale.py b/gpMgmt/bin/el8_migrate_locale/el8_migrate_locale.py index 78ae50284367..fbab24b96f22 100644 --- a/gpMgmt/bin/el8_migrate_locale/el8_migrate_locale.py +++ b/gpMgmt/bin/el8_migrate_locale/el8_migrate_locale.py @@ -1,5 +1,8 @@ #!/usr/bin/env python #!-*- coding: utf-8 -*- +from __future__ import print_function +from builtins import range +from builtins import object import argparse import sys from pygresql.pg import DB @@ -14,7 +17,7 @@ import re try: from pygresql import pg -except ImportError, e: +except ImportError as e: sys.exit('ERROR: Cannot import modules. Please check that you have sourced greengage_path.sh. Detail: ' + str(e)) class connection(object): @@ -40,10 +43,10 @@ def get_port_from_conf(self): file = datadir +'/postgresql.conf' if os.path.isfile(file): with open(file) as f: - for line in f.xreadlines(): - match = re.search('port=\d+',line) + for line in f: + match = re.search(r'port=\d+',line) if match: - match1 = re.search('\d+', match.group()) + match1 = re.search(r'\d+', match.group()) if match1: return match1.group() @@ -113,21 +116,21 @@ def dump_index_info(self, fn): # print all catalog indexes that might be affected. cindex = self.get_affected_catalog_indexes() if cindex: - print>>f, "\c ", self.dbname + print("\\c ", self.dbname, file=f) for indexrelid, indexname, tablename, collname, indexdef in cindex: - print>>f, "-- catalog indexrelid:", indexrelid, "| index name:", indexname, "| table name:", tablename, "| collname:", collname, "| indexdef: ", indexdef - print>>f, self.handle_one_index(indexname) - print>>f + print("-- catalog indexrelid:", indexrelid, "| index name:", indexname, "| table name:", tablename, "| collname:", collname, "| indexdef: ", indexdef, file=f) + print(self.handle_one_index(indexname), file=f) + print(file=f) # print all user indexes in all databases that might be affected. for dbname in dblist: index = self.get_affected_user_indexes(dbname) if index: - print>>f, "\c ", dbname + print("\\c ", dbname, file=f) for indexrelid, indexname, tablename, collname, indexdef in index: - print>>f, "-- indexrelid:", indexrelid, "| index name:", indexname, "| table name:", tablename, "| collname:", collname, "| indexdef: ", indexdef - print>>f, self.handle_one_index(indexname) - print>>f + print("-- indexrelid:", indexrelid, "| index name:", indexname, "| table name:", tablename, "| collname:", collname, "| indexdef: ", indexdef, file=f) + print(self.handle_one_index(indexname), file=f) + print(file=f) f.close() @@ -215,7 +218,7 @@ def get_affected_partitioned_tables(self, dbname): result = tabsForDebug.getresult() logger.debug("There are {} range partitioning tables in database {}.".format(len(result), dbname)) if len(result): - print tabsForDebug + print(tabsForDebug) # Filtered partition range table that partition key in collate types. filterTabs = db.query(sql) @@ -223,7 +226,7 @@ def get_affected_partitioned_tables(self, dbname): if len(filterResult): logger.warning("There are {} range partitioning tables with partition key in collate types(like varchar, char, text) in database {}, these tables might be affected due to Glibc upgrade and should be checked when doing OS upgrade from EL7 to EL8.".format(len(filterResult), dbname)) if self.loglevel == logging.DEBUG: - print filterTabs + print(filterTabs) db.close() return filterResult @@ -238,7 +241,7 @@ def get_custom_opclass_as_distribute_keys_tables(self, dbname): result = tables.getresult() if result: logger.warning("There are {} tables in database {} that the distribution key is using custom operator class, should be checked when doing OS upgrade from EL7 to EL8.".format(len(result), dbname)) - print tables + print(tables) db.close() # Escape double-quotes in a string, so that the resulting string is suitable for @@ -306,7 +309,7 @@ def dump_tables(self, fn): msg, size = self.get_table_size_info(dbname, parrelid) table_info.append((parrelid, tablename, coll, attname, msg, size)) # if no default partition, give a warning, in case of migrate failed - if has_default_partition == 'f': + if not has_default_partition: logger.warning("no default partition for {}".format(tablename)) else: # start multiple threads to check if the rows are still in the correct partitions after os upgrade, if check failed, add these tables to filtertabs @@ -319,9 +322,9 @@ def dump_tables(self, fn): # dump the table info to the specified output file if table_info: - print>>f, "-- order table by size in %s order " % 'ascending' if self.order_size_ascend else '-- order table by size in descending order' - print>>f, "\c ", dbname - print>>f + print("-- order table by size in %s order " % 'ascending' if self.order_size_ascend else '-- order table by size in descending order', file=f) + print("\\c ", dbname, file=f) + print(file=f) # sort the tables by size if self.order_size_ascend: @@ -335,9 +338,9 @@ def dump_tables(self, fn): coll = result[2] attname = result[3] msg = result[4] - print>>f, "-- parrelid:", parrelid, "| coll:", coll, "| attname:", attname, "| msg:", msg - print>>f, self.handle_one_table(name) - print>>f + print("-- parrelid:", parrelid, "| coll:", coll, "| attname:", attname, "| msg:", msg, file=f) + print(self.handle_one_table(name), file=f) + print(file=f) # print the total partition table size self.print_size_summary_info() @@ -345,7 +348,7 @@ def dump_tables(self, fn): f.close() def print_size_summary_info(self): - print "---------------------------------------------" + print("---------------------------------------------") KB = float(1024) MB = float(KB ** 2) GB = float(KB ** 3) @@ -360,7 +363,7 @@ def print_size_summary_info(self): print("total partition tables : {}".format(self.total_roots)) print("total leaf partitions : {}".format(self.total_leafs)) - print "---------------------------------------------" + print("---------------------------------------------") # start multiple threads to do the check def concurrent_check(self, dbname): @@ -432,7 +435,7 @@ def check_partitiontables_by_guc(self, idx, dbname): self.filtertabs.append((parrelid, tablename, coll, attname, msg, size)) self.filtertabslock.release() has_error = False - if has_default_partition == 'f': + if not has_default_partition: logger.warning("no default partition for {}".format(tablename)) db.query("set gp_detect_data_correctness = 0;") @@ -461,8 +464,8 @@ def parse_inputfile(self): with open(self.script_file) as f: for line in f: sql = line.strip() - if sql.startswith("\c"): - db_name = sql.split("\c")[1].strip() + if sql.startswith("\\c"): + db_name = sql.split("\\c")[1].strip() if (sql.startswith("reindex") and sql.endswith(";") and sql.count(";") == 1): self.dbdict[db_name].append(sql) if (sql.startswith("begin;") and sql.endswith("commit;")): @@ -470,7 +473,7 @@ def parse_inputfile(self): def run(self): try: - for db_name, commands in self.dbdict.items(): + for db_name, commands in list(self.dbdict.items()): total_counts = len(commands) logger.info("db: {}, total have {} commands to execute".format(db_name, total_counts)) for command in commands: @@ -496,7 +499,7 @@ def run_alter_command(self, db_name, command): db.query(analyze_sql) db.close() - except Exception, e: + except Exception as e: logger.error("{}".format(str(e))) def parseargs(): diff --git a/gpMgmt/bin/gpactivatestandby b/gpMgmt/bin/gpactivatestandby index edfd5bf36e27..0a6f6205ba1f 100755 --- a/gpMgmt/bin/gpactivatestandby +++ b/gpMgmt/bin/gpactivatestandby @@ -13,6 +13,7 @@ failed. Will promote the standby, update the system catalog tables to make the standby master instance as the new master. """ +from builtins import range import os import sys import signal @@ -31,7 +32,7 @@ try: from gppylib.gplog import get_default_logger, setup_tool_logging, enable_verbose_logging, get_logger_if_verbose from gppylib import gparray from gppylib.userinput import ask_yesno -except ImportError, e_: +except ImportError as e_: sys.exit('ERROR: Cannot import modules. Please check that you ' 'have sourced greengage_path.sh. Detail: ' + str(e_)) @@ -211,7 +212,7 @@ def check_original_master_status(options): recovery_conf_file_location = os.path.join(options.master_data_dir, GP_RECOVERY_CONF_FILE) if not os.path.exists(recovery_conf_file_location): - logger.warn('Please make sure the command gpactivatestandby is executed on current Standby host') + logger.warning('Please make sure the command gpactivatestandby is executed on current Standby host') raise GpActivateStandbyException('Critical required file on standby \"%s\" is not present.' % recovery_conf_file_location) with open(recovery_conf_file_location) as read_file: @@ -228,11 +229,11 @@ def check_original_master_status(options): cmd = pg.ReadPostmasterTempFile.remote('Read postmaster file', original_master_port, original_master_hostname) (file_exists, _, _) = cmd.getResults() if file_exists: - logger.warn('Appears that there is an active postgres process on %s port=%d' % (original_master_hostname, original_master_port)) - logger.warn('Need to stop current master before activating the standby.') - logger.warn('Rare possibility is just /tmp/.s.PGSQL.%d and /tmp/.s.PGSQL.%d.* files exist on %s.' % (original_master_port, original_master_port, original_master_hostname)) - logger.warn('Need to delete these files') - logger.warn('Then call this utility again.') + logger.warning('Appears that there is an active postgres process on %s port=%d' % (original_master_hostname, original_master_port)) + logger.warning('Need to stop current master before activating the standby.') + logger.warning('Rare possibility is just /tmp/.s.PGSQL.%d and /tmp/.s.PGSQL.%d.* files exist on %s.' % (original_master_port, original_master_port, original_master_hostname)) + logger.warning('Need to delete these files') + logger.warning('Then call this utility again.') raise GpActivateStandbyException('Active postgres process on master') #------------------------------------------------------------------------- @@ -321,7 +322,7 @@ def promote_standby(master_data_dir): cmd.run(validateAfter=True) logger.debug('Waiting for connection...') - for i in xrange(600): + for i in range(600): try: dburl = dbconn.DbURL() with dbconn.connect(dburl, utility=True, logConn=False) as conn: @@ -331,7 +332,7 @@ def promote_standby(master_data_dir): dbconn.execSQL(conn, 'CHECKPOINT') logger.info('Standby master is promoted') return True - except pygresql.InternalError, e: + except pygresql.InternalError as e: pass time.sleep(1) @@ -393,7 +394,7 @@ try: else: sys.exit(0) -except Exception, e_: +except Exception as e_: logger.fatal('Error activating standby master: %s' % str(e_)) sys.exit(2) diff --git a/gpMgmt/bin/gpcheckcat b/gpMgmt/bin/gpcheckcat index 5188b13a5509..504078dfbaa7 100755 --- a/gpMgmt/bin/gpcheckcat +++ b/gpMgmt/bin/gpcheckcat @@ -21,12 +21,26 @@ Usage: gpcheckcat [