Skip to content

Commit

Permalink
Merge pull request #4688 from bcgov/dev-marshal-DV-merge
Browse files Browse the repository at this point in the history
Updates from test-marshal to dev-marshal
  • Loading branch information
aparna-aot authored Nov 1, 2023
2 parents 3ada012 + 89a936d commit 6fb0eee
Show file tree
Hide file tree
Showing 167 changed files with 12,353 additions and 637 deletions.
12 changes: 6 additions & 6 deletions .github/workflows/katalon-ci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ jobs:
runs-on: windows-latest
steps:
- name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v3.0
with:
ref: dev-automationscripts
- name: set screen resolution
Expand All @@ -35,7 +35,7 @@ jobs:
dir
shell: cmd
- name: Katalon Studio Github Action
uses: katalon-studio/katalon-studio-github-action@v2
uses: katalon-studio/katalon-studio-github-action@v3.0
with:
version: '8.2.0'
projectPath: '${{ github.workspace }}/testing/foi-qa-automation/foi-qa-automation.prj'
Expand All @@ -50,15 +50,15 @@ jobs:
dir
shell: cmd
- name: Setup Node.js
uses: actions/setup-node@v2
uses: actions/setup-node@v3.0
with:
node-version: '14'
- name: Create collection report
run: |
npm i -g xunit-viewer
xunit-viewer -r artifacts\JUnit_Report.xml -o artifacts\foi-test.html
- name: Archive Katalon report
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v3.0
with:
name: katalon-report
path: artifacts
Expand All @@ -67,9 +67,9 @@ jobs:
# runs-on: macos-latest
# steps:
# - name: Checkout
# uses: actions/checkout@v2
# uses: actions/checkout@v3
# - name: Katalon Studio Github Action
# uses: katalon-studio/katalon-studio-github-action@v2.1
# uses: katalon-studio/katalon-studio-github-action@v3.2.0
# with:
# version: '7.5.5'
# projectPath: '${{ github.workspace }}'
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ request-management-api/request_api/__pycache__/__init__.cpython-38.pyc
request-management-api/request_api/models/__pycache__/*.*
request-management-api/migrations/__pycache__/*.*
request-management-api/.env
notification-manager/env/*

*.pyc
*/__pycache__/*
Expand Down Expand Up @@ -92,6 +93,7 @@ axisintegrationapi/MCS.FOI.AXISIntegration/MCS.FOI.AXISIntegrationWebAPI/*.user
axisintegrationapi/MCS.FOI.AXISIntegration/MCS.FOI.AXISIntegrationWebAPI/Properties/PublishProfiles/FolderProfile.pubxml
axisintegrationapi/MCS.FOI.AXISIntegration/MCS.FOI.AXISIntegrationWebAPI/Properties/PublishProfiles/FolderProfile.pubxml.user
apps/
datamigrations/FOIMOD.CFD.ETL.DataMigration/.vs/FOIMOD.CFD.ETL.DataMigration/v16/.suo

datamigrations/FOIMOD.CFD.ConsoleApp.DocMigration/*/bin/*
datamigrations/FOIMOD.CFD.DocMigration.AXIS.DAL/*/bin/*
Expand Down
53 changes: 53 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,59 @@ Freedom of Information modernization.

## Installation

### For mac
#### Run the microservices in docker
1. Clone this repo
2. Place the appropriate .env file to the root folder of entire repo
3. Change docker-compose.yml line 152 from windows.Dockerfile to mac.Dockerfile
4. Make sure you are logged into the VPN if working remotely
5. Compose up the docker-compose.yml file in the root folder by either right-clicking and selecting 'Compose up' in VS Code or running the command
```docker compose -f "docker-compose.yml" up -d --build```

#### Add IP address to hosts on local system if accessing remotely
1. Log into vpn
2. Click statistics icon (bottom left of AnyConnect, the graph icon)
3. Note down client address (IPv4)
4. In your terminal run the command ``` sudo nano /etc/hosts ```
5. Add the ip address from above to the list, with alias value ``` foiflow.local ```
6. Save and exit

#### API performance issues
The app (particularly the API) may run very slowly. Performance can be improved by turning off Redis caching, commenting out the following code in [KeycloakAdminService](https://github.com/bcgov/foi-flow/blob/main/request-management-api/request_api/services/external/keycloakadminservice.py) (at ```foi-flow/request-management-api/request_api/services/external/keycloakadminservice.py```). Make sure that you don't commit these changes.

Comment out the following, and change the _accesstoken value to None
```diff
def get_token(self):
_accesstoken=None
try:
+ #cache_client = redis.from_url(self.cache_redis_url,decode_responses=True)
- cache_client = redis.from_url(self.cache_redis_url,decode_responses=True)
+ _accesstoken = None
- _accesstoken = cache_client.get("foi:kcsrcacnttoken")
if _accesstoken is None:
url = '{0}/auth/realms/{1}/protocol/openid-connect/token'.format(self.keycloakhost,self.keycloakrealm)
params = {

'client_id': self.keycloakclientid,
'grant_type': 'password',
'username' : self.keycloakadminserviceaccount,
'password': self.keycloakadminservicepassword,
'client_secret':self.keycloakclientsecret
}
x = requests.post(url, params, verify=True).content.decode('utf-8')
_accesstoken = str(ast.literal_eval(x)['access_token'])
+ #cache_client.set("foi:kcsrcacnttoken",_accesstoken,ex=int(self.kctokenexpiry))
- cache_client.set("foi:kcsrcacnttoken",_accesstoken,ex=int(self.kctokenexpiry))
except BusinessException as exception:
print("Error happened while accessing token on KeycloakAdminService {0}".format(exception.message))
+ #finally:
+ #cache_client = None
- finally:
- cache_client = None
return _accesstoken
```
Again, make sure you don't commit these changes.

## Project Status
The project is in the very early stages of development. The codebase will be changing frequently.

Expand Down
50 changes: 50 additions & 0 deletions apps/forms-flow-ai/forms-flow-bpm/mac.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Modified by Yichun Zhao and Walter Moar

# Maven build
FROM artifacts.developer.gov.bc.ca/docker-remote/maven:3.6.1-jdk-11-slim AS MAVEN_TOOL_CHAIN

RUN apt-get update \
&& apt-get install -y git

ARG FORMIO_SOURCE_REPO_BRANCH=v4.0.5-alpha
ARG FORMIO_SOURCE_REPO_URL=https://github.com/AOT-Technologies/forms-flow-ai.git

RUN git clone -b ${FORMIO_SOURCE_REPO_BRANCH} ${FORMIO_SOURCE_REPO_URL} /bpm/

#RUN cp /bpm/forms-flow-bpm/pom-docker.xml /tmp/pom.xml
#RUN cp /bpm/forms-flow-bpm/settings-docker.xml /usr/share/maven/ref/
COPY ./pom-docker.xml /tmp/pom.xml
COPY ./settings-docker.xml /usr/share/maven/ref/

WORKDIR /tmp/

# This allows Docker to cache most of the maven dependencies
RUN mvn -s /usr/share/maven/ref/settings-docker.xml dependency:resolve-plugins dependency:resolve dependency:go-offline -B
RUN rm -rf /bpm/forms-flow-bpm/src/main/resources/processes
RUN cp -r /bpm/forms-flow-bpm/src/ /tmp/src/

ARG CUSTOM_SRC_DIR=src/main/

# Override these files they have custom changes in the sbc_divapps directory
COPY ./${CUSTOM_SRC_DIR}/ /tmp/${CUSTOM_SRC_DIR}/
RUN mvn -s /usr/share/maven/ref/settings-docker.xml package -Dmaven.test.skip



# Final custom slim java image (for apk command see jdk-11.0.3_7-alpine-slim)
FROM --platform=amd64 artifacts.developer.gov.bc.ca/docker-remote/adoptopenjdk/openjdk11:jdk-11.0.3_7-alpine

ENV JAVA_VERSION jdk-11.0.3+7
ENV JAVA_HOME=/opt/java/openjdk \
PATH="/opt/java/openjdk/bin:$PATH"

EXPOSE 8080
# OpenShift has /app in the image, but it's missing when doing local development - Create it when missing
RUN test ! -d /app && mkdir /app || :
# Add spring boot application
RUN mkdir -p /app
COPY --from=MAVEN_TOOL_CHAIN /tmp/target/forms-flow-bpm.jar ./app
RUN chmod a+rwx -R /app
WORKDIR /app
VOLUME /tmp
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app/forms-flow-bpm.jar"]
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ public AXISRequest GetAXISRequest(string request)
axisRequest.StartDate = RequestsHelper.ConvertDateToString(row, "requestProcessStart", "yyyy-MM-dd");
axisRequest.DueDate = RequestsHelper.ConvertDateToString(row, "dueDate", "yyyy-MM-dd");
axisRequest.CFRDueDate = RequestsHelper.ConvertDateToString(row, "cfrDueDate", "yyyy-MM-dd");
axisRequest.OriginalDueDate = RequestsHelper.ConvertDateToString(row, "originalDueDate", "yyyy-MM-dd");

axisRequest.DeliveryMode = RequestsHelper.GetDeliveryMode(Convert.ToString(row["deliveryMode"]));
axisRequest.ReceivedMode = RequestsHelper.GetReceivedMode(Convert.ToString(row["receivedMode"]));
Expand Down Expand Up @@ -83,6 +84,7 @@ public AXISRequest GetAXISRequest(string request)
axisRequest.Ispiiredacted = true;
axisRequest.RequestPageCount = Convert.ToInt32(row["requestPageCount"]);
axisRequest.SubjectCode = Convert.ToString(row["subjectCode"]);
axisRequest.IdentityVerified = Convert.ToString(row["identityVerified"]);
List<Ministry> ministryList = new()
{
new Ministry(RequestsHelper.GetMinistryCode(Convert.ToString(row["selectedMinistry"])))
Expand Down Expand Up @@ -151,6 +153,7 @@ private DataTable GetAxisRequestData(string request)
when requests.IREQUESTID = redaction.IREQUESTID and redaction.IDOCID = ldocuments.IDOCID then ldocuments.SIPAGECOUNT
else 0 end) as requestPageCount,
REPLACE(requestfields.CUSTOMFIELD33, CHAR(160), ' ') as subjectCode,
requestfields.CUSTOMFIELD75 as identityVerified,
(SELECT TOP 1 cfr.sdtDueDate FROM tblRequestForDocuments cfr WITH (NOLOCK)
INNER JOIN tblProgramOffices programoffice WITH (NOLOCK) ON programoffice.tiProgramOfficeID = cfr.tiProgramOfficeID
WHERE requests.iRequestID = cfr.iRequestID
Expand Down Expand Up @@ -182,7 +185,7 @@ LEFT OUTER JOIN dbo.TBLREQUESTCUSTOMFIELDS requestfields WITH (NOLOCK) ON reques
requesters.vcAddress1, requesters.vcAddress2, requesters.vcCity, requesters.vcZipCode,
requesters.vcHome, requesters.vcMobile, requesters.vcWork1, requesters.vcWork2, requesters.vcFirstName, requesters.vcLastName, requesters.vcMiddleName,
requests.iRequestID, requesters.vcCompany, requesters.vcEmailID, onbehalf.vcFirstName, onbehalf.vcLastName, onbehalf.vcMiddleName,
requestTypes.iLabelID, requests.vcVisibleRequestID, requests.tiOfficeID, office.OFFICE_ID,requestorfields.CUSTOMFIELD35, REPLACE(requestfields.CUSTOMFIELD33, CHAR(160), ' ')";
requestTypes.iLabelID, requests.vcVisibleRequestID, requests.tiOfficeID, office.OFFICE_ID,requestorfields.CUSTOMFIELD35, REPLACE(requestfields.CUSTOMFIELD33, CHAR(160), ' '),requestfields.CUSTOMFIELD75";
DataTable dataTable = new();
using (sqlConnection = new SqlConnection(ConnectionString))
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,9 @@ public class AXISRequest
[DataMember(Name = "Extensions")]
public List<Extension> Extensions { get; set; }

[DataMember(Name = "identityVerified")]
public string IdentityVerified { get; set; }

}

[DataContract]
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 17
VisualStudioVersion = 17.4.33213.308
MinimumVisualStudioVersion = 10.0.40219.1
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.ConsoleApp.DocMigration", "FOIMOD.CFD.ConsoleApp.DocMigration\FOIMOD.CFD.ConsoleApp.DocMigration.csproj", "{F3EE4F36-2422-4F11-ADEA-8455BA9B7A27}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.AXIS.DAL", "FOIMOD.CFD.DocMigration.DAL\FOIMOD.CFD.DocMigration.AXIS.DAL.csproj", "{F63AE4FE-892B-40EB-8D88-3BC3433A0FF5}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.S3Uploader", "FOIMOD.CFD.DocMigration.S3Uploader\FOIMOD.CFD.DocMigration.S3Uploader.csproj", "{59E9C03A-DE05-48ED-8B16-252CC23DB5E6}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.FOIFLOW.DAL", "FOIMOD.CFD.DocMigration.FOIFLOW.DAL\FOIMOD.CFD.DocMigration.FOIFLOW.DAL.csproj", "{F2A928AC-15B5-4124-95D1-E7706000EFB4}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.BAL", "FOIMOD.CFD.DocMigration.BAL\FOIMOD.CFD.DocMigration.BAL.csproj", "{FF7CB66C-60A6-4130-919A-3DBECB869CAE}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.Utils", "FOIMOD.CFD.DocMigration.Utils\FOIMOD.CFD.DocMigration.Utils.csproj", "{68B638B6-9118-48CE-AC5A-A1D00265B653}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.Models", "FOIMOD.CFD.DocMigration.Models\FOIMOD.CFD.DocMigration.Models.csproj", "{AC053E49-6F19-4F5D-AF91-DA6CFD4C4D74}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.Utils.UnitTests", "FOIMOD.CFD.DocMigration.Utils.UnitTests\FOIMOD.CFD.DocMigration.Utils.UnitTests.csproj", "{9D27E50C-7209-4BDB-9379-627A7CD9B2FA}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FOIMOD.CFD.DocMigration.AXIS.DAL.UnitTests", "FOIMOD.CFD.DocMigration.AXIS.DAL.UnitTests\FOIMOD.CFD.DocMigration.AXIS.DAL.UnitTests.csproj", "{4402E37D-BA96-4C9C-9AD2-78A866D9C2E6}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "FOIMOD.CFD.DocMigration.FOIFLOW.DAL.Tests", "FOIMOD.CFD.DocMigration.FOIFLOW.DAL.Tests\FOIMOD.CFD.DocMigration.FOIFLOW.DAL.Tests.csproj", "{E964523D-91C1-4E1C-860F-52E57E826E16}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{F3EE4F36-2422-4F11-ADEA-8455BA9B7A27}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{F3EE4F36-2422-4F11-ADEA-8455BA9B7A27}.Debug|Any CPU.Build.0 = Debug|Any CPU
{F3EE4F36-2422-4F11-ADEA-8455BA9B7A27}.Release|Any CPU.ActiveCfg = Release|Any CPU
{F3EE4F36-2422-4F11-ADEA-8455BA9B7A27}.Release|Any CPU.Build.0 = Release|Any CPU
{F63AE4FE-892B-40EB-8D88-3BC3433A0FF5}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{F63AE4FE-892B-40EB-8D88-3BC3433A0FF5}.Debug|Any CPU.Build.0 = Debug|Any CPU
{F63AE4FE-892B-40EB-8D88-3BC3433A0FF5}.Release|Any CPU.ActiveCfg = Release|Any CPU
{F63AE4FE-892B-40EB-8D88-3BC3433A0FF5}.Release|Any CPU.Build.0 = Release|Any CPU
{59E9C03A-DE05-48ED-8B16-252CC23DB5E6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{59E9C03A-DE05-48ED-8B16-252CC23DB5E6}.Debug|Any CPU.Build.0 = Debug|Any CPU
{59E9C03A-DE05-48ED-8B16-252CC23DB5E6}.Release|Any CPU.ActiveCfg = Release|Any CPU
{59E9C03A-DE05-48ED-8B16-252CC23DB5E6}.Release|Any CPU.Build.0 = Release|Any CPU
{F2A928AC-15B5-4124-95D1-E7706000EFB4}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{F2A928AC-15B5-4124-95D1-E7706000EFB4}.Debug|Any CPU.Build.0 = Debug|Any CPU
{F2A928AC-15B5-4124-95D1-E7706000EFB4}.Release|Any CPU.ActiveCfg = Release|Any CPU
{F2A928AC-15B5-4124-95D1-E7706000EFB4}.Release|Any CPU.Build.0 = Release|Any CPU
{FF7CB66C-60A6-4130-919A-3DBECB869CAE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{FF7CB66C-60A6-4130-919A-3DBECB869CAE}.Debug|Any CPU.Build.0 = Debug|Any CPU
{FF7CB66C-60A6-4130-919A-3DBECB869CAE}.Release|Any CPU.ActiveCfg = Release|Any CPU
{FF7CB66C-60A6-4130-919A-3DBECB869CAE}.Release|Any CPU.Build.0 = Release|Any CPU
{68B638B6-9118-48CE-AC5A-A1D00265B653}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{68B638B6-9118-48CE-AC5A-A1D00265B653}.Debug|Any CPU.Build.0 = Debug|Any CPU
{68B638B6-9118-48CE-AC5A-A1D00265B653}.Release|Any CPU.ActiveCfg = Release|Any CPU
{68B638B6-9118-48CE-AC5A-A1D00265B653}.Release|Any CPU.Build.0 = Release|Any CPU
{AC053E49-6F19-4F5D-AF91-DA6CFD4C4D74}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{AC053E49-6F19-4F5D-AF91-DA6CFD4C4D74}.Debug|Any CPU.Build.0 = Debug|Any CPU
{AC053E49-6F19-4F5D-AF91-DA6CFD4C4D74}.Release|Any CPU.ActiveCfg = Release|Any CPU
{AC053E49-6F19-4F5D-AF91-DA6CFD4C4D74}.Release|Any CPU.Build.0 = Release|Any CPU
{9D27E50C-7209-4BDB-9379-627A7CD9B2FA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{9D27E50C-7209-4BDB-9379-627A7CD9B2FA}.Debug|Any CPU.Build.0 = Debug|Any CPU
{9D27E50C-7209-4BDB-9379-627A7CD9B2FA}.Release|Any CPU.ActiveCfg = Release|Any CPU
{9D27E50C-7209-4BDB-9379-627A7CD9B2FA}.Release|Any CPU.Build.0 = Release|Any CPU
{4402E37D-BA96-4C9C-9AD2-78A866D9C2E6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{4402E37D-BA96-4C9C-9AD2-78A866D9C2E6}.Debug|Any CPU.Build.0 = Debug|Any CPU
{4402E37D-BA96-4C9C-9AD2-78A866D9C2E6}.Release|Any CPU.ActiveCfg = Release|Any CPU
{4402E37D-BA96-4C9C-9AD2-78A866D9C2E6}.Release|Any CPU.Build.0 = Release|Any CPU
{E964523D-91C1-4E1C-860F-52E57E826E16}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{E964523D-91C1-4E1C-860F-52E57E826E16}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E964523D-91C1-4E1C-860F-52E57E826E16}.Release|Any CPU.ActiveCfg = Release|Any CPU
{E964523D-91C1-4E1C-860F-52E57E826E16}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {ED380BD1-F2BB-40F6-A8D3-12308661A924}
EndGlobalSection
EndGlobal
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net7.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<PlatformTarget>x86</PlatformTarget>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="3.7.201.12" />
<PackageReference Include="Microsoft.Extensions.Configuration" Version="7.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.2" />
<PackageReference Include="System.Data.Odbc" Version="7.0.0" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\FOIMOD.CFD.DocMigration.BAL\FOIMOD.CFD.DocMigration.BAL.csproj" />
<ProjectReference Include="..\FOIMOD.CFD.DocMigration.Models\FOIMOD.CFD.DocMigration.Models.csproj" />
</ItemGroup>
<ItemGroup>
<None Update="appsettings.dev.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
<None Update="appsettings.json">
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
</None>
</ItemGroup>
</Project>
Loading

0 comments on commit 6fb0eee

Please sign in to comment.