Changes

Summary

  1. updated deployment spec for the aggregation workflows on PROD (details)
  2. added deployment spec for the resolution workflow on PROD (details)
Commit 9276bdfcd3b7a2b5be16a94f28f6a27fee21b241 by Claudio Atzori
updated deployment spec for the aggregation workflows on PROD
The file was modifiedPROD_aggregation/Jenkinsfile
Commit 6ea22867e2a5ec5bc9236e854ee19ff632c72574 by Claudio Atzori
added deployment spec for the resolution workflow on PROD
The file was modifiedPROD/Jenkinsfile

Summary

  1. Datacite: Code Refactor generated a general SparkApplication Scala where all the spark scala have to inherit (details)
  2. add test resources (details)
  3. remove working dir after test (details)
  4. [DOIBOOST Process] fix filtering to filter results with non null id (details)
  5. minor fix (details)
  6. Sprint 4 (details)
  7. added sprint3,4, removed 2, chaos (details)
  8. Sprint 4 (details)
  9. Added indi_funder_country_collab (details)
  10. code formatting (details)
  11. [resolution wf] added optional parameter to skip the entity resolution (details)
  12. added test to verify the relation inverse lookup operation (details)
  13. code formatting (details)
  14. [cleaning wf] fixed OAF record navigation, a mapping defined on a container object would have prevented the natvigation to continue on its properties (details)
  15. minor: fixed hive action name (details)
  16. added stats computation on the graph hive DB tables (details)
  17. implemented Instance serialization features requested in https://support.openaire.eu/issues/7156 (details)
  18. avoid to filter OAF records defined as invisible = true (details)
Commit 2164a2a8898da176799bab3815244b234746eb45 by Sandro La Bruzzo
Datacite: Code Refactor generated a general SparkApplication Scala where all the spark scala have to inherit
Commented a little the Datacite transformation code
The file was addeddhp-common/src/main/java/eu/dnetlib/dhp/application/AbstractScalaApplication.scala
The file was modifieddhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/datacite/DataciteToOAFTransformation.scala
The file was modifieddhp-workflows/dhp-aggregation/src/test/java/eu/dnetlib/dhp/datacite/DataciteToOAFTest.scala
The file was addeddhp-common/src/main/java/eu/dnetlib/dhp/application/SparkScalaApplication.scala
The file was modifieddhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/datacite/GenerateDataciteDatasetSpark.scala
The file was addeddhp-workflows/dhp-aggregation/src/main/java/eu/dnetlib/dhp/datacite/DataciteModelConstants.scala
The file was modifieddhp-workflows/dhp-aggregation/src/main/resources/log4j.properties
Commit 028a8acad8b392673c9c94e49f15dca1c51b279a by Sandro La Bruzzo
add test resources
The file was addeddhp-workflows/dhp-aggregation/src/test/resources/eu/dnetlib/dhp/actionmanager/datacite/dataset/part-00000.parquet
Commit feea154e89bc24d0b408c6927230b4987a6c4357 by Sandro La Bruzzo
remove working dir after test
The file was modifieddhp-workflows/dhp-aggregation/src/test/java/eu/dnetlib/dhp/datacite/DataciteToOAFTest.scala
Commit 5fd0e610bf1cb1cd6f8e80e047602fba2703ddce by Miriam Baglioni
[DOIBOOST Process] fix filtering to filter results with non null id
The file was modifieddhp-workflows/dhp-doiboost/src/main/java/eu/dnetlib/doiboost/mag/SparkProcessMAG.scala
Commit 1e1f5e4fe08be8bc81d23e1c1d40d0886fef00f4 by Sandro La Bruzzo
minor fix
The file was removeddhp-common/src/main/java/eu/dnetlib/dhp/application/AbstractScalaApplication.scala
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/application/SparkScalaApplication.scala
The file was modifieddhp-common/pom.xml
Commit 29f69f2f89d55213b2e07aa618b9113299c76939 by dpierrakos
Sprint 4
The file was modifieddhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step16-createIndicatorsTables.sql
Commit 0b4163ee0b3bec77bafe8811b7ee4f7f2a02a7cb by Antonis Lempesis
added sprint3,4, removed 2, chaos
The file was modifieddhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step20-createMonitorDB.sql
Commit 25fc8abf77c512e650164972ba5cc5ade366b112 by dpierrakos
Sprint 4
The file was modifieddhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step16-createIndicatorsTables.sql
Commit 09fc2afdcad627ec47cedfabf18e946d4036a0db by dpierrakos
Added indi_funder_country_collab
Kept only indi_pub_has_cc_licence
The file was modifieddhp-workflows/dhp-stats-update/src/main/resources/eu/dnetlib/dhp/oa/graph/stats/oozie_app/scripts/step16-createIndicatorsTables.sql
Commit 5c6d32853713276f7808b3b3dcff046a447a1017 by Claudio Atzori
code formatting
The file was modifieddhp-workflows/dhp-graph-mapper/src/test/java/eu/dnetlib/dhp/oa/graph/clean/GraphCleaningFunctionsTest.java
Commit 014e872ae1d00a4cf32506e1ab2e204874913e25 by Claudio Atzori
[resolution wf] added optional parameter to skip the entity resolution
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/resolution/oozie_app/workflow.xml
Commit 01e5e0142ac9d0e181e1d62b03c564bf4c15953d by Claudio Atzori
added test to verify the relation inverse lookup operation
The file was addeddhp-workflows/dhp-blacklist/src/test/java/eu/dnetlib/dhp/blacklist/BlacklistRelationTest.java
Commit 4fe788881796525044517c19d633ff1df017656f by Claudio Atzori
code formatting
The file was modifieddhp-workflows/dhp-blacklist/src/test/java/eu/dnetlib/dhp/blacklist/BlacklistRelationTest.java
Commit d85af6fc255acf12347b7424bb9d596d63abeefa by Claudio Atzori
[cleaning wf] fixed OAF record navigation, a mapping defined on a container object would have prevented the natvigation to continue on its properties
The file was modifieddhp-workflows/dhp-graph-mapper/src/test/resources/eu/dnetlib/dhp/oa/graph/clean/relation.json
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/clean/OafCleaner.java
The file was modifieddhp-workflows/dhp-graph-mapper/src/test/java/eu/dnetlib/dhp/oa/graph/clean/GraphCleaningFunctionsTest.java
The file was modifieddhp-workflows/dhp-graph-mapper/src/test/resources/eu/dnetlib/dhp/oa/graph/clean/synonyms.txt
Commit cfa456076908aca3a828bec81a34bf5a40788f2a by Claudio Atzori
minor: fixed hive action name
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/hive/oozie_app/workflow.xml
Commit 3b19821f3c357f37f91467caad84a6f7cf874c34 by Claudio Atzori
added stats computation on the graph hive DB tables
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/hive/oozie_app/lib/scripts/postprocessing.sql
Commit 9cac283bec9abc1394c6866c8c164a773af1a52f by Claudio Atzori
implemented Instance serialization features requested in https://support.openaire.eu/issues/7156
The file was addeddhp-workflows/dhp-graph-provision/src/main/java/eu/dnetlib/dhp/oa/provision/model/XmlInstance.java
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/publication.json
The file was modifieddhp-workflows/dhp-graph-provision/src/main/resources/eu/dnetlib/dhp/oa/provision/template/instance.st
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/XmlRecordFactoryTest.java
The file was modifieddhp-workflows/dhp-graph-provision/src/main/java/eu/dnetlib/dhp/oa/provision/utils/TemplateFactory.java
The file was modifieddhp-workflows/dhp-graph-provision/src/main/java/eu/dnetlib/dhp/oa/provision/utils/XmlRecordFactory.java
Commit 863a2f9db3497b5d2f8263f61adec878c51c8cba by Claudio Atzori
avoid to filter OAF records defined as invisible = true
The file was modifieddhp-workflows/dhp-graph-mapper/src/test/java/eu/dnetlib/dhp/oa/graph/clean/GraphCleaningFunctionsTest.java
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
The file was addeddhp-workflows/dhp-graph-mapper/src/test/resources/eu/dnetlib/dhp/oa/graph/clean/result_invisible.json
The file was addeddhp-workflows/dhp-graph-mapper/src/test/resources/eu/dnetlib/dhp/oa/graph/clean/result_missing_invisible.json