apex-source-control/design_docs.md

The problem:

There is a common practice of not placing databases or back-end applications under source-control management (SCM). APEX is web application development tool designed for back-end developers with limited front-end experience so the culture of ignoring SCM permeates into the development process. This tool aims to make integrating APEX apps with SCM a painless process.

System requirements for use:

• A Linux or Mac OSX environment
• Downloaded APEX 5.0 files or above
• An export APEX_HOME with $APEX_HOME/utilities/oracle/apex/*.class being the location of the above mentioned files
• An installation of Oracle Instant Client
• An export ORACLE_HOME with an existing $ORACLE_HOME/jdbc/lib/jdbc6.jar (likely included in installation of Oracle Instant Client)
• sqlplus on path

What users should already know:

• How to use any SCM tool such as git or subversion
• How to use APEX (obviously)
• Very basic understanding of the command line

Tools/APIs we use:

Here are the pre-existing tools we used to help make these scripts

APEXExport & APEXExportSplitter

No docs for these classes but here is a quick overview discovered through trial & error and research: Export allows you to export an app from APEX into a large SQL file using the command line, and Splitter splits that large SQL file into multiple files which fit into a nice directory structure.

apex_application_install

Again there are no docs but read this blog post by Joel Kallman from Oracle on the tool http://joelkallman.blogspot.ca/2010/07/apexapplicationinstall.html. A quick overview is that essentially this comes with a bunch of pre-packaged functionality useful for installing from a file(s) generated by APEXExport & APEXExportSplitter into APEX.

Npm Scripts

Used as a front end runner for our scripts read: https://docs.npmjs.com/cli/run-script.

Scripts we create…

For each script we talk about 1. its functionality and 2. How it works. And occasionally why it’s needed if not immediately obvious.

(Note: there are more scripts than this but those are mostly for style and calling from other scripts and not listed here)

...for dealing with APEX:

• file-to-apex
• Converts a file directory generated by APEXExport(Splitter) into an APEX app
• Uses apex_application_install to set import config data (specified in the user’s config file) such as app_alias, workspace_id, and application_id
• apex-to-file
• Converts an APEX app into files
• Checks for proper config and then calls APEXExport & APEXExportSplitter
• uninstall-apex
• Uninstalls your app from APEX
• Calls the delete_application and end_environment scripts generated by Splitting the file generated by APEXExport

...for user convenience:

• new-conf-file
• Creates a new config file based on user input
• Reduces chances of typos in config file creation and was all around convenient during the development process
• Gathers user input for each config value and writes it to a file
• switch-conf-file
• Switches the config file being used
• Was convenient and time-saving during development so I figured it would be nice to expose this functionality to the user as well
• A config file is referenced by an asc.conf symlink, this script lists the available configs and changes the symlink to point to the new file of the user’s choice
• generate-app-id
• Generates an application ID that is unused in your copy of APEX
• Solves the problem of accidentally overwriting another developers app since an unused app id can always be generated
• Exposes the apex_application_install.generate_application_id functionality to the user
• read-conf-file
• States which config file is being used and lists the contents
• Again it was nice to type a single short command rather than 2 longer ones.
• readlink asc.conf && cat asc.conf

Problems faced during creation and our solutions:

1) Application Object ids change on every export causing large conflicts in source control

We use APEX's -expOrginalIds option, and since APEX 5.0 seems to have a mechanism for dealing with conflicting Ids (tested by importing several applications using the same Object Ids) we can import into Oracle using the original Ids (is this problem even one that the user needs to know about? Since the problem basically just comes from the paper but turns out to not be an issue it seems as if we just created an issue out of nothing)

2) Can’t import into APEX without workspace_id

(Believe me, just figuring out that this was the problem without any proper docs was an entire problem of its own) We use PL/SQL to find the workspace_id (using the user input for workspace) and set the app’s workspace_id as such. Solved in file-to-apex script.

3) Possibility of overwriting another developer’s app by picking the same app_id

We create a script which automatically generates an unused app_id by exposing the functionality of apex_application_install.generate_application_id. Solved in generate-app-id.

4) App-aliases must be unique across workspace

We auto generate the app_alias using the app_id. Since the app_id must be unique this ensures that the app_alias will also be unique. We use the string ‘F’ + app_id as the alias. Solved in file-to-apex script.

5) When importing to certain servers (like prod) we want to have control over the app_alias

We add an optional app_alias value to the config file. If filled in we set the alias to the given value, otherwise we use the auto generated one described previously. Solved in file-to-apex script.

6) How to generalize (parameterize) scripts without causing source-control noise

We create config files and introduce a symlink which points to the config file which needs to be used. We then warn the user to ignore the symlink (and probably the other config files) in order to remove all possible source-control noise. Now the only noise should come the APEX app itself, that is, we are not adding any extra noise ourselves with these scripts.

7) There exists the possibility of overwriting work

Further explanation: If I were to make changes in APEX and then, without running apex-to-file, pull from the main SCM repo and run file-to-apex I would lose all my changes without any warning. Solution: The current solution is to encourage proper workflow practices with users, especially frequent committing. This falls under the “user ought to know how to use SCM” responsibility. Further Improvement: It would be possible to, after pulling from the main SCM repo run a apex-to-file, check the two versions against each other, and force the user to merge any conflicts properly.

8) Code is not easily sharable and could result in many different versions of the same code causing confusion and conflicting usage

We use npm to package the scripts as a node module.

Unsolved issues/future improvement: 1) Random noise in app meta-data

To reproduce:

• Have two developers working on their own version of an app
• Developer 1 makes no change, runs apex-to-file, commits, and pushes to SCM
• Developer 2 makes no changes and pulls from SCM to incorporate Developer 1’s changes
• There are now conflicts shown in the merge attempt, most likely in ./apex/application/create_application.sql and some randomly generated empty sql files, despite none actually existing.

Why it occurs:

Every APEX app comes with meta-data; such as who last edited the app and when, what workspace ID was this last edited in, etc.; this obviously changes with every edit so SCM will always detect conflicts on every single attempt to merge.

How to avoid/fix:

There is no easy avoidal mechanism or fix for this issue. One possible solution is to write scripts which detect garbage conflicts and automatically deal with them; however, while some conflicts are consistent and predictable, others occur seemingly at random.

2) Deleting a page will always win merges

To reproduce:

• Have two developers working on their own versions of an app (i.e. working on different features to be added).
• Developer 1 deletes a page in the app and pushes to the project's repo.
• Developer 2 merges developer 1's changes and decides to keep the deleted page in the merge.
• When developer 2 reinstalls the app into apex the page will still be deleted.

Why it occurs:

Since a page was deleted, it's removed from the install.sql script generated by apex. When git sees the merge of the two developers, only one has changed the install.sql script so it automatically applies those changes to the merge meaning the page is no longer called during install so it's never created in your apex application.

How to avoid/fix:

Deleting a page from an app is an important decision in any development process thus you should consult with your team before doing so. If you do end up deleting a page you will have to manually re-add the install lines the install lines to the install.sql script (found in /your/project/dir/apex/install.sql).

3) Config files on top project level are kind of gross

It might be a good idea to keep the config files in a place where the user cannot easily see them. Before this can be implemented however we would need to create an edit-conf-file function so the user can interact with config files without ever actually needing to touch one of them (all “touching” done through our provided scripts).