Changes

Summary

  1. enrichment with subworkflows (details)
Commit cc40b46ff64a56c7039e62cfe0e6785cb501a1a9 by Claudio Atzori
enrichment with subworkflows
The file was modifiedBETA/Jenkinsfile

Summary

  1. [graph cleaning] added cleaning for result.publisher and result.instance.license (details)
  2. fixed doiboost process workflow, removed references to the ProcessORCID step (details)
  3. Extracted the correct original type to pass to instanceTypeMapping in Crossref Mapping (details)
  4. code formatting (details)
  5. [graph grouping] added isLookupUrl to the workflow definition, passed to the grouping spark aciton (details)
  6. avoid NPEs in Vocabulary.getTermBySynonym (details)
  7. avoid NPEs (details)
  8. avoid NPEs (details)
  9. [bulktagging] fixed workflow parameters (details)
  10. [community_organization propagation] fixed workflow parameters (details)
  11. added serialization for the new fields imported for the Irish tender (details)
  12. [dedup] added isLookupUrl to the graph consistency workflow definition, required now by the entity grouping phase (details)
  13. [orcid enrichment] fixed workflow definition (details)
  14. first version of the workflow single step (details)
  15. [bulktagging] setting first step of bulktaggin as the copy of the entities and relations not involved in the tagging' (details)
  16. [community_result_propagation] adjusting starting poit of workflow (details)
  17. [enrichment] passing the community API base URL (details)
  18. logging typo (details)
  19. [graph cleaning] avoid stack overflow error when navigating Oaf objects declaring an Enum (details)
  20. code formatting (details)
  21. adjusting workflow definition (details)
  22. removed not needed parameter (details)
  23. [graph provision] added tests for the new model fields (details)
  24. [cleaning] allow enriched orcids to pass the cleaning, rule out non-orcid author pids (details)
  25. code formatting (details)
  26. [graph provision] added tests for new peerreviewed field (details)
  27. - (details)
  28. [doiboost - preprocess] remove transition to orcid preparation from sequence of steps at the beginning of the workflow (details)
  29. - (details)
  30. updated the transformation Baseline workflow to include mdstore rollback/commit action (details)
  31. uploaded input parameters on CreateBaseline WF (details)
  32. added needed parameter (details)
  33. - (details)
  34. refactoring after compiletion (details)
  35. added metaresourcetype to the result hive DB view (details)
  36. adjustments for country propagation (details)
  37. adding the bulkTag parameter file in the folder for the oozie workflow for bulkTagging. Changes the path in the class (details)
  38. changed the path to the parameter file in the class for entitytoorganization propagation (details)
  39. added properties file in the forlder for the workflow of orcid propagation. Changes the path in the classes implementing the propagationchanged the path to the parameter file in the class for entitytoorganization propagation (details)
  40. changed in the classes the path for the property files for the propagation of community from project (details)
  41. added properties file in the forlder for the workflow of project to result propagation. Changes the path in the classes implementing the propagation (details)
  42. added properties file in the forlder for the workflow of result to community from organization propagation. Changes the path in the classes implementing the propagation (details)
  43. added properties file in the forlder for the workflow of result to community from semrel propagation. Changes the path in the classes implementing the propagation (details)
  44. added properties file in the forlder for the workflow of result to organization from inst repo propagation. Changes the path in the classes implementing the propagation (details)
  45. SparkCreateSimRels: (details)
  46. Do no longer use dedupId information from pivotHistory Database (details)
  47. Generate "merged" dedup id relations also for records that are filtered out by the cut parameters (details)
  48. Use dedup_wf_002 in place of dedup_wf_001 to make explicit a different algorithm has been used to generate those kind of ids (details)
  49. Create dedup record for "merged" pivots (details)
  50. refined mapping for the extraction of the original resource type (details)
Commit 1ba582de3c9625417032294cd9212400d4e12663 by Claudio Atzori
[graph cleaning] added cleaning for result.publisher and result.instance.license
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
Commit 511a98dd800d2697134b0c01f36f7b09f2caa8f1 by Claudio Atzori
fixed doiboost process workflow, removed references to the ProcessORCID step
The file was modifieddhp-workflows/dhp-doiboost/src/main/resources/eu/dnetlib/dhp/doiboost/process/oozie_app/workflow.xml
Commit 3caf6ff27ef91aa6b684fdb91ea4d7e1274f9b3d by Sandro La Bruzzo
Extracted the correct original type to pass to instanceTypeMapping in Crossref Mapping
The file was modifieddhp-workflows/dhp-doiboost/src/main/scala/eu/dnetlib/doiboost/crossref/Crossref2Oaf.scala
Commit 178a14c491ea24847d02e3eb2295a08198b37d9b by Claudio Atzori
code formatting
The file was modifieddhp-workflows/dhp-doiboost/src/main/scala/eu/dnetlib/doiboost/crossref/Crossref2Oaf.scala
Commit 2a233a89aa49c2ec59c95ba86d256ab46eb3db9b by Claudio Atzori
[graph grouping] added isLookupUrl to the workflow definition, passed to the grouping spark aciton
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/group/oozie_app/workflow.xml
Commit e6086efc53804cf3e29d29fd6b2f1b8f958f9130 by Claudio Atzori
avoid NPEs in Vocabulary.getTermBySynonym
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/common/vocabulary/Vocabulary.java
Commit 74b185d07b3655f6e964ce5ae0610695d1685903 by Claudio Atzori
avoid NPEs
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
Commit 7c3041b276c2c27f235f68e3ebdc4ae3b862dd5b by Claudio Atzori
avoid NPEs
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
Commit 3c3bdb83187d572c101447def9e5f75febe8efa0 by Claudio Atzori
[bulktagging] fixed workflow parameters
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/bulktag/SparkBulkTagJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/input_bulkTag_parameters.json
Commit c5b72531301c4317ebc63261e1148b10c1c6f240 by Claudio Atzori
[community_organization propagation] fixed workflow parameters
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromorganization/oozie_app/workflow.xml
Commit 321922772b71c70e074eb97efb46d920f1031752 by Claudio Atzori
added serialization for the new fields imported for the Irish tender
The file was modifieddhp-workflows/dhp-graph-provision/src/main/java/eu/dnetlib/dhp/oa/provision/utils/XmlRecordFactory.java
Commit 431c6bb08a2713df4cb1dbe0abeb60011c409349 by Claudio Atzori
[dedup] added isLookupUrl to the graph consistency workflow definition, required now by the entity grouping phase
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/consistency/oozie_app/workflow.xml
Commit 259c69e446c855a3f3da8cfec532f2ac0e2ff87e by Claudio Atzori
[orcid enrichment] fixed workflow definition
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/enrich/orcid/oozie_app/workflow.xml
Commit 616622d2bb90f4a5e43a9da2dc864ef03dab1f00 by Miriam Baglioni
first version of the workflow single step
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromorganization/oozie_app/config-default.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfrominstrepo/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/entitytoorganizationfromsemrel/oozie_app/config-default.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromsemrel/oozie_app/config-default.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromsemrel/input_preparecommunitytoresult_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/entitytoorganizationfromsemrel/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/bulktag/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/entitytoorganizationfromsemrel/input_propagation_parameter.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfromsemrel/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromproject/input_preparecommunitytoresult_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromorganization/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromproject/input_communitytoresult_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromproject/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/oozie_app/config-default.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/api/Utils.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromsemrel/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromsemrel/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/entitytoorganizationfromsemrel/input_preparation_parameter.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfrominstrepo/oozie_app/config-default.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromproject/oozie_app/config-default.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromsemrel/SparkResultToCommunityThroughSemRelJob.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/oozie_app/import.txt
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromproject/PrepareResultCommunitySet.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromsemrel/PrepareResultCommunitySetStep1.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/oozie_app/config-default.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/projecttoresult/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfromsemrel/oozie_app/config-default.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/SparkCountryPropagationJob.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/oozie_app/config-default.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/PrepareDatasourceCountryAssociation.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/projecttoresult/oozie_app/config-default.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromorganization/SparkResultToCommunityFromOrganizationJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromproject/SparkResultToCommunityFromProject.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/bulktag/SparkBulkTagJob.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/bulktag/oozie_app/config-default.xml
Commit c0cde53bf647d00bf5d633ef0ef906b4984d35b5 by Miriam Baglioni
[bulktagging] setting first step of bulktaggin as the copy of the entities and relations not involved in the tagging'
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/oozie_app/workflow.xml
Commit 336fb31d8778cbfbb75a555165b55ee48518a9ba by Miriam Baglioni
[community_result_propagation] adjusting starting poit of workflow
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromorganization/oozie_app/workflow.xml
Commit c381bacee01b7f5a3918e0e77d4a55d8e969c04b by Claudio Atzori
[enrichment] passing the community API base URL
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromproject/oozie_app/workflow.xml
Commit 70eb1796b20c1001d4c49655e72dbbe0c791252a by Claudio Atzori
logging typo
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromproject/PrepareResultCommunitySet.java
Commit cb71a7936bb98f498693fe75335c60947f5cbef6 by Claudio Atzori
[graph cleaning] avoid stack overflow error when navigating Oaf objects declaring an Enum
The file was addeddhp-workflows/dhp-graph-mapper/src/test/resources/eu/dnetlib/dhp/oa/graph/clean/graph/orp/orp.json
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/clean/OafCleaner.java
The file was modifieddhp-workflows/dhp-graph-mapper/src/test/java/eu/dnetlib/dhp/oa/graph/clean/CleanGraphSparkJobTest.java
Commit aba95ed1d17a1683729ebddf2afff785bcb5e138 by Claudio Atzori
code formatting
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
Commit d4eedada71436a7cae1a5ab154598503b8f36e91 by Miriam Baglioni
adjusting workflow definition
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/SparkCountryPropagationJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromorganization/SparkResultToCommunityFromOrganizationJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromproject/SparkResultToCommunityFromProject.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromsemrel/SparkResultToCommunityThroughSemRelJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/PrepareDatasourceCountryAssociation.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/bulktag/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/api/Utils.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromsemrel/PrepareResultCommunitySetStep1.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/job.properties
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/bulktag/SparkBulkTagJob.java
Commit 8752d275fae9bc7764cd2ee049b6321d44b70528 by Miriam Baglioni
removed not needed parameter
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/orcidtoresultfromsemrel/input_orcidtoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/orcidtoresultfromsemrel/SparkOrcidToResultFromSemRelJob.java
Commit 7e8eff40c1660d5d277fdc36ff87d2e264e6549d by Claudio Atzori
[graph provision] added tests for the new model fields
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/XmlRecordFactoryTest.java
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/solr/conf/exploreTestConfig/managed-schema
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/XmlIndexingJobTest.java
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/solr/conf/testConfig/managed-schema
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/publication.json
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/fields.xml
Commit 84d54643cf55d86d42cea236569c84a1b2c339bc by Claudio Atzori
[cleaning] allow enriched orcids to pass the cleaning, rule out non-orcid author pids
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
Commit 98cce5bfb22c5d761fbf7044834959efdec4e31e by Claudio Atzori
code formatting
The file was modifieddhp-common/src/main/java/eu/dnetlib/dhp/schema/oaf/utils/GraphCleaningFunctions.java
Commit ff924215b8e457b3e05e98792db0c6b3bdd0fa0d by Claudio Atzori
[graph provision] added tests for new peerreviewed field
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/solr/conf/exploreTestConfig/managed-schema
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/XmlRecordFactoryTest.java
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/IndexRecordTransformerTest.java
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/fields.xml
The file was modifieddhp-workflows/dhp-graph-provision/src/test/resources/eu/dnetlib/dhp/oa/provision/solr/conf/testConfig/managed-schema
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/XmlIndexingJobTest.java
Commit 0d8e496a6317943a28282ffdd0ee5a4d735f61f7 by Miriam Baglioni
-
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/SparkResultToOrganizationFromIstRepoJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/input_prepareresultorg_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/AppendNewRelations.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/PrepareResultInstRepoAssociation.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/job.properties
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/orcidtoresultfromsemrel/SparkOrcidToResultFromSemRelJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfrominstrepo/oozie_app/workflow.xml
Commit 01ce0b9c7626cb853acd218db011a498cb8d06e2 by Miriam Baglioni
[doiboost - preprocess] remove transition to orcid preparation from sequence of steps at the beginning of the workflow
The file was modifieddhp-workflows/dhp-doiboost/src/main/resources/eu/dnetlib/dhp/doiboost/preprocess/oozie_app/workflow.xml
Commit 3eca5d2e1c302a7427ffa735c95ac96a6419caec by Miriam Baglioni
-
The file was addeddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/entitytoorganizationfromsemrel/SparkEntityToOrganizationFromSemRel.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/entitytoorganizationfromsemrel/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/entitytoorganizationfromsemrel/SparkResultToOrganizationFromSemRel.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/entitytoorganizationfromsemrel/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/entitytoorganizationfromsemrel/StepActions.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/job.properties
The file was modifieddhp-workflows/dhp-enrichment/src/test/java/eu/dnetlib/dhp/entitytoorganizationfromsemrel/SparkJobTest.java
Commit 9d342a47da489d71c3a739b06943a4f09a1225ee by Sandro La Bruzzo
updated the transformation Baseline workflow to include mdstore rollback/commit action
The file was modifieddhp-workflows/dhp-aggregation/src/main/scala/eu/dnetlib/dhp/sx/bio/ebi/SparkCreateBaselineDataFrame.scala
The file was modifieddhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/sx/bio/pubmed/oozie_app/workflow.xml
Commit 15fd93a2b66f6829cfac0b1350266664371d1df5 by Sandro La Bruzzo
uploaded input parameters on CreateBaseline WF
The file was modifieddhp-workflows/dhp-aggregation/src/main/resources/eu/dnetlib/dhp/sx/bio/ebi/baseline_to_oaf_params.json
Commit d410ea8a4176341cdebaa76179c77b5fdd45c631 by Miriam Baglioni
added needed parameter
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/AppendNewRelations.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/input_newrelation_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/job.properties
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/entitytoorganizationfromsemrel/oozie_app/workflow.xml
Commit 4740c808f735193e8975f09e1a6841eb4d9a676f by Miriam Baglioni
-
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromproject/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/projecttoresult/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromsemrel/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromorganization/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfromsemrel/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/entitytoorganizationfromsemrel/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/orcidtoresultfromsemrel/input_prepareorcidtoresult_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromproject/input_preparecommunitytoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromproject/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/orcidtoresultfromsemrel/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromorganization/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/input_prepareresultorg_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromsemrel/input_preparecommunitytoresult_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/countrypropagation/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/input_newrelation_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/orcidtoresultfromsemrel/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/projecttoresult/input_projecttoresult_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/projecttoresult/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfrominstrepo/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/projecttoresult/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/input_bulkTag_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/entitytoorganizationfromsemrel/input_preparation_parameter.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/countrypropagation/input_prepareresultcountry_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/input_propagationresulaffiliationfrominstrepo_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/input_eosc_bulkTag_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromproject/input_communitytoresult_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/projecttoresult/input_prepareprojecttoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/PrepareDatasourceCountryAssociation.java
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/input_eoscTag_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromorganization/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromsemrel/oozie_app/config-default.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/bulktag/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/job.properties
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/entitytoorganizationfromsemrel/input_propagation_parameter.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/entitytoorganizationfromsemrel/oozie_app/workflow.xml
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/entitytoorganizationfromsemrel/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfromsemrel/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/countrypropagation/input_prepareassoc_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/orcidtoresultfromsemrel/input_prepareorcidtoresult_parameters2.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromsemrel/input_communitytoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/main/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromorganization/input_preparecommunitytoresult_parameters.json
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/hostedbymap/oozie_app/download.sh
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/countrypropagation/input_countrypropagation_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromorganization/input_communitytoresult_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromsemrel/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/orcidtoresultfromsemrel/input_orcidtoresult_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromproject/oozie_app/config-default.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/countrypropagation/oozie_app/workflow.xml
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/bulktag/datasourcemaster_parameters.json
The file was removeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/resulttocommunityfromsemrel/input_preparecommunitytoresult2_parameters.json
Commit 5011c4d11a4c3884c99d784ed31a336ba89f8bfc by Miriam Baglioni
refactoring after compiletion
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/XmlIndexingJobTest.java
The file was modifieddhp-workflows/dhp-graph-provision/src/test/java/eu/dnetlib/dhp/oa/provision/IndexRecordTransformerTest.java
Commit 62104790ae63d08946f0e340d67f3182cb469b8d by Claudio Atzori
added metaresourcetype to the result hive DB view
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/resources/eu/dnetlib/dhp/oa/graph/hive/oozie_app/lib/scripts/postprocessing.sql
Commit 3afd4aa57bb107e35f71108c64c45ada698cf8a7 by Miriam Baglioni
adjustments for country propagation
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/PrepareResultCountrySet.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/entitytoorganizationfromsemrel/PrepareInfo.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/oozie_app/workflow.xml
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/input_prepareassoc_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/PrepareDatasourceCountryAssociation.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/input_prepareresultcountry_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/countrypropagation/SparkCountryPropagationJob.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/countrypropagation/input_countrypropagation_parameters.json
Commit b06aea0adfe716fede41a6fd38e847dc90dd4692 by Miriam Baglioni
adding the bulkTag parameter file in the folder for the oozie workflow for bulkTagging. Changes the path in the class
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/bulktag/SparkBulkTagJob.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/bulktag/input_bulkTag_parameters.json
Commit 89f269c7f4b63070358724213b5d39fac0678916 by Miriam Baglioni
changed the path to the parameter file in the class for entitytoorganization propagation
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/entitytoorganizationfromsemrel/SparkEntityToOrganizationFromSemRel.java
Commit 009730b3d1616fa3337cad380b9ff8e55641c9a5 by Miriam Baglioni
added properties file in the forlder for the workflow of orcid propagation. Changes the path in the classes implementing the propagationchanged the path to the parameter file in the class for entitytoorganization propagation
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/input_orcidtoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/orcidtoresultfromsemrel/SparkOrcidToResultFromSemRelJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/orcidtoresultfromsemrel/PrepareResultOrcidAssociationStep2.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/input_prepareorcidtoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/orcidtoresultfromsemrel/PrepareResultOrcidAssociationStep1.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/orcidtoresultfromsemrel/input_prepareorcidtoresult_parameters2.json
Commit f2352e8a78017f26f297833546e1a0853c5a89b7 by Miriam Baglioni
changed in the classes the path for the property files for the propagation of community from project
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromproject/PrepareResultCommunitySet.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromproject/SparkResultToCommunityFromProject.java
Commit 2f7b9ad815358857dd14656ae1e4b160e7721662 by Miriam Baglioni
added properties file in the forlder for the workflow of project to result propagation. Changes the path in the classes implementing the propagation
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/projecttoresult/PrepareProjectResultsAssociation.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/projecttoresult/input_prepareprojecttoresult_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/projecttoresult/input_projecttoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/projecttoresult/SparkResultToProjectThroughSemRelJob.java
Commit 2f3b5a133d4ddfc4ed6a38366c927330d2c25b08 by Miriam Baglioni
added properties file in the forlder for the workflow of result to community from organization propagation. Changes the path in the classes implementing the propagation
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromorganization/input_communitytoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromorganization/PrepareResultCommunitySet.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromorganization/SparkResultToCommunityFromOrganizationJob.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromorganization/input_preparecommunitytoresult_parameters.json
Commit 9f966b59d446ba83d9dd002dddaf1d9585a3b037 by Miriam Baglioni
added properties file in the forlder for the workflow of result to community from semrel propagation. Changes the path in the classes implementing the propagation
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromsemrel/input_communitytoresult_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromsemrel/input_preparecommunitytoresult_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromsemrel/SparkResultToCommunityThroughSemRelJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromsemrel/PrepareResultCommunitySetStep2.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttocommunityfromsemrel/input_preparecommunitytoresult2_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttocommunityfromsemrel/PrepareResultCommunitySetStep1.java
Commit cb14470ba6779bd6f5dea3e1b937512295c0854a by Miriam Baglioni
added properties file in the forlder for the workflow of result to organization from inst repo propagation. Changes the path in the classes implementing the propagation
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/PrepareResultInstRepoAssociation.java
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfrominstrepo/input_newrelation_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfrominstrepo/input_prepareresultorg_parameters.json
The file was addeddhp-workflows/dhp-enrichment/src/main/resources/eu/dnetlib/dhp/wf/subworkflows/resulttoorganizationfrominstrepo/input_propagationresulaffiliationfrominstrepo_parameters.json
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/SparkResultToOrganizationFromIstRepoJob.java
The file was modifieddhp-workflows/dhp-enrichment/src/main/java/eu/dnetlib/dhp/resulttoorganizationfrominstrepo/AppendNewRelations.java
Commit 02636e802c26c284efa1415d168815c5b23ed655 by Giambattista Bloisi
SparkCreateSimRels:
- Create dedup blocks from the complete queue of records matching cluster key instead of truncating the results
- Clean titles once before clustering and similarity comparisons
- Added support for filtered fields in model
- Added support for sorting List fields in model
- Added new JSONListClustering and numAuthorsTitleSuffixPrefixChain clustering functions
- Added new maxLengthMatch comparator function
- Use reduced complexity Levenshtein with threshold in levensteinTitle
- Use reduced complexity AuthorsMatch with threshold early-quit
- Use incremental Connected Component to decrease comparisons in similarity match in BlockProcessor
- Use new clusterings configuration in Dedup tests

SparkWhitelistSimRels: use left semi join for clarity and performance

SparkCreateMergeRels:
- Use new connected component algorithm that converge faster than Spark GraphX provided algorithm
- Refactored to use Windowing sorting rather than groupBy to reduce memory pressure
- Use historical pivot table to generate singleton rels, merged rels and keep continuity with dedupIds used in the past
- Comparator for pivot record selection now uses "tomorrow" as filler for missing or incorrect date instead of "2000-01-01"
- Changed generation of ids of type dedup_wf_001 to avoid collisions

DedupRecordFactory: use reduceGroups instead of mapGroups to decrease memory pressure
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/model/SparkDeduper.scala
The file was modifieddhp-workflows/dhp-dedup-openaire/src/test/java/eu/dnetlib/dhp/oa/dedup/SparkDedupTest.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/SortedNgramPairs.java
The file was modifiedpom.xml
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/scan/oozie_app/config-default.xml
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/PersonClustering.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/model/FieldDef.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/DedupRecordFactory.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/AbstractSparkAction.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/ImmutableFieldValue.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/util/PaceResolver.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/NgramPairs.java
The file was addeddhp-workflows/dhp-dedup-openaire/src/test/resources/eu/dnetlib/dhp/dedup/pivot_history/pivot_history.json
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/SuffixPrefix.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/common/AbstractPaceFunctions.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/util/MapDocumentUtil.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/PersonHash.java
The file was addeddhp-workflows/dhp-dedup-openaire/src/main/scala/com/kwartile/lib/cc/ConnectedComponent.scala
The file was addeddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/JSONListClustering.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/SparkWhitelistSimRels.java
The file was removeddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/graph/GraphProcessor.scala
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/tree/InstanceTypeMatch.java
The file was addeddhp-pace-core/src/main/java/eu/dnetlib/pace/tree/MaxLengthMatch.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/test/resources/eu/dnetlib/dhp/dedup/conf/sw.curr.conf.json
The file was addeddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/NumAuthorsTitleSuffixPrefixChain.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/IdGenerator.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/SpaceTrimmingFieldValue.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/UrlClustering.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/WordsSuffixPrefix.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/test/resources/eu/dnetlib/dhp/dedup/conf/orp.curr.conf.json
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/model/Identifier.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/WordsStatsSuffixPrefixChain.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/createCC_parameters.json
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/util/BlockProcessor.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/LowercaseClustering.java
The file was modifieddhp-pace-core/src/test/java/eu/dnetlib/pace/clustering/ClusteringFunctionTest.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/tree/AuthorsMatch.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/tree/LevensteinTitle.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/ClusteringFunction.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/Acronyms.java
The file was removeddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/graph/ConnectedComponent.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/RandomClusteringFunction.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/SparkCreateMergeRels.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/resources/eu/dnetlib/dhp/oa/dedup/scan/oozie_app/workflow.xml
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/LastNameFirstInitial.java
The file was addeddhp-pace-core/src/main/java/eu/dnetlib/pace/util/IncrementalConnectedComponents.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/model/SparkModel.scala
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/Ngrams.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/tree/support/AbstractComparator.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/test/resources/eu/dnetlib/dhp/dedup/conf/pub.curr.conf.json
The file was addeddhp-pace-core/src/test/java/eu/dnetlib/pace/util/IncrementalConnectedComponentsTest.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/KeywordsClustering.java
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/model/ClusteringDef.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/test/resources/eu/dnetlib/dhp/dedup/conf/ds.curr.conf.json
The file was modifieddhp-pace-core/src/main/java/eu/dnetlib/pace/clustering/AbstractClusteringFunction.java
Commit 1287315ffb546397bcbcac588fd5b80a62cab665 by Giambattista Bloisi
Do no longer use dedupId information from pivotHistory Database
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/SparkCreateMergeRels.java
Commit 831cc1fddececffc80701931ad9dab4d9926192b by Giambattista Bloisi
Generate "merged" dedup id relations also for records that are filtered out by the cut parameters
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/SparkCreateMergeRels.java
Commit 10e135db1eb26cf6383d02f2318c8e6701631553 by Giambattista Bloisi
Use dedup_wf_002 in place of dedup_wf_001 to make explicit a different algorithm has been used to generate those kind of ids
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/IdGenerator.java
Commit 3c66e3bd7bd7fbe14f068b5176ae3681e941fda9 by Giambattista Bloisi
Create dedup record for "merged" pivots
Do not create dedup records for group that have more than 20 different acceptance date
The file was modifieddhp-workflows/dhp-dedup-openaire/src/test/java/eu/dnetlib/dhp/oa/dedup/SparkDedupTest.java
The file was modifieddhp-workflows/dhp-dedup-openaire/src/main/java/eu/dnetlib/dhp/oa/dedup/DedupRecordFactory.java
Commit 2753044d13da0465b8b9061e70252ed6ac69a325 by Claudio Atzori
refined mapping for the extraction of the original resource type
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/raw/OafToOafMapper.java
The file was modifieddhp-workflows/dhp-graph-mapper/src/main/java/eu/dnetlib/dhp/oa/graph/raw/OdfToOafMapper.java