Merge pull request #447 from vimleshmishra/master

vimleshmishra · SumoSourabh · Terraform User · web-flow · commit 93e5fbfb92b1 · 2022-08-10T15:04:52.000+05:30
* Updating code owners for the github repository (#439) Updating code owners Co-authored-by: Sourabh Jain <sourabh.jain@delphix.com> * Create CodeOwners File [Terraform] * CP-8174 VSDK Changes to support source sizing decorators (#441) * __hash__ is needed in python 3 * DLPX-79760 SnapshotDefinitionDataPaths object is not hashable since py2-3 transition * DOC-2234 Update new logo in developer.delphix.com * Resolved COnflicts * revert logo chnages * revert logo chnages * Resolve Conflicts * revert changes * revert logo.png * Resolve Conflicts * DOC-2234 Update new logo in developer.delphix.com * logo chnages * DOC-2234 Update new logo in developer.delphix.com * new favicon * incorporated review comments * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * Changes suggested by Aleks * CP-8174 VSDK Changes to support source sizing decorators * reverted dvp version related changes * reverted extra comma * update engine API version * Fixed Formatting Issues * Fixed Formatting Issues * Fixed Unit Tests * Fixed Unit Tests * CP-8293 Update version to 4.0.5.dev0 for virtualization-sdk develop branch before updating github actions (#442) * __hash__ is needed in python 3 * DLPX-79760 SnapshotDefinitionDataPaths object is not hashable since py2-3 transition * DOC-2234 Update new logo in developer.delphix.com * Resolved COnflicts * revert logo chnages * revert logo chnages * Resolve Conflicts * revert changes * revert logo.png * Resolve Conflicts * DOC-2234 Update new logo in developer.delphix.com * logo chnages * DOC-2234 Update new logo in developer.delphix.com * new favicon * incorporated review comments * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * Changes suggested by Aleks * CP-8174 VSDK Changes to support source sizing decorators * reverted dvp version related changes * reverted extra comma * update engine API version * Fixed Formatting Issues * Fixed Formatting Issues * Fixed Unit Tests * Fixed Unit Tests * Bumping up dvp-api, vsdk * uppdate formatting * Revert "CP-8293 Update version to 4.0.5.dev0 for virtualization-sdk develop branch before updating github actions (#442)" (#443) This reverts commit 222517e. * CP-8293 Update version to 4.0.5.dev0 for virtualization-sdk develop branch before updating github actions (#444) * __hash__ is needed in python 3 * DLPX-79760 SnapshotDefinitionDataPaths object is not hashable since py2-3 transition * DOC-2234 Update new logo in developer.delphix.com * Resolved COnflicts * revert logo chnages * revert logo chnages * Resolve Conflicts * revert changes * revert logo.png * Resolve Conflicts * DOC-2234 Update new logo in developer.delphix.com * logo chnages * DOC-2234 Update new logo in developer.delphix.com * new favicon * incorporated review comments * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * DLPX-81195 develop version of vSDK needs to be bumped up to 4.0.4 for testing to use latest vSDK code * Changes suggested by Aleks * CP-8174 VSDK Changes to support source sizing decorators * reverted dvp version related changes * reverted extra comma * update engine API version * Fixed Formatting Issues * Fixed Formatting Issues * Fixed Unit Tests * Fixed Unit Tests * Bumping up dvp-api, vsdk * uppdate formatting * for rerun the failed runs * bumpversion support dev bumping * revert bumpversion support dev bumping * bump up version from dev0 to dev1 * bump up version from dev0 to dev1 * virtualization-sdk master branch will always refer dlpx-appgate stage/release * virtualization-sdk master branch will always refer dlpx-appgate stage/release * virtualization-sdk master branch will always refer dlpx-appgate stage/release * virtualization-sdk master branch will always refer dlpx-appgate stage/release * virtualization-sdk master branch will always refer dlpx-appgate stage/release * Fixed Issue: Lint common - Python38 Co-authored-by: Sourabh Jain <45626610+SumoSourabh@users.noreply.github.com> Co-authored-by: Sourabh Jain <sourabh.jain@delphix.com> Co-authored-by: Terraform User <terraform@delphix.com>
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 4.0.4
+current_version = 4.0.5.dev1
 commit = False
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\.(?P<release>[a-z]+)(?P<dev>\d+))?
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
@@ -1 +1,5 @@
-*       @ankursarin @nhlien93 @mothslaw
+#
+# Copyright (c) 2022 by Delphix. All rights reserved.
+#
+
+* @delphix/gatekeepers-appdata
diff --git a/common/setup.py b/common/setup.py
@@ -4,7 +4,7 @@
 PYTHON_SRC = 'src/main/python'
 
 install_requires = [
-  "dvp-api == 1.6.3",
+  "dvp-api == 1.7.0.dev0",
 ]
 
 with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file:
diff --git a/common/src/main/python/dlpx/virtualization/common/VERSION b/common/src/main/python/dlpx/virtualization/common/VERSION
@@ -1 +1 @@
-4.0.4
+4.0.5.dev1
diff --git a/common/src/main/python/dlpx/virtualization/common/_common_classes.py b/common/src/main/python/dlpx/virtualization/common/_common_classes.py
@@ -65,7 +65,8 @@ def user(self):
         return self.__user
 
     def to_proto(self):
-        """Converts plugin class RemoteConnection to protobuf class common_pb2.RemoteConnection
+        """Converts plugin class RemoteConnection to protobuf class
+        common_pb2.RemoteConnection
         """
         remote_connection = common_pb2.RemoteConnection()
         remote_connection.environment.CopyFrom(self.environment.to_proto())
@@ -74,7 +75,8 @@ def to_proto(self):
 
     @staticmethod
     def from_proto(connection):
-        """Converts protobuf class common_pb2.RemoteConnection to plugin class RemoteConnection
+        """Converts protobuf class common_pb2.RemoteConnection to plugin class
+        RemoteConnection
         """
         if not isinstance(connection, common_pb2.RemoteConnection):
             raise IncorrectTypeError(
@@ -411,7 +413,8 @@ def public_key(self):
 
     @staticmethod
     def from_proto(credentials_result):
-        """Converts protobuf class libs_pb2.CredentialsResult to plugin class KeyPairCredentials
+        """Converts protobuf class libs_pb2.CredentialsResult to plugin class
+        KeyPairCredentials
         """
         if not isinstance(credentials_result, libs_pb2.CredentialsResult):
             raise IncorrectTypeError(
diff --git a/dvp/src/main/python/dlpx/virtualization/VERSION b/dvp/src/main/python/dlpx/virtualization/VERSION
@@ -1 +1 @@
-4.0.4
+4.0.5.dev1
diff --git a/libs/setup.py b/libs/setup.py
@@ -7,7 +7,7 @@
     version = version_file.read().strip()
 
 install_requires = [
-  "dvp-api == 1.6.3",
+  "dvp-api == 1.7.0.dev0",
   "dvp-common == {}".format(version)
 ]
 
diff --git a/libs/src/main/python/dlpx/virtualization/libs/VERSION b/libs/src/main/python/dlpx/virtualization/libs/VERSION
@@ -1 +1 @@
-4.0.4
+4.0.5.dev1
diff --git a/platform/setup.py b/platform/setup.py
@@ -7,7 +7,7 @@
     version = version_file.read().strip()
 
 install_requires = [
-  "dvp-api == 1.6.3",
+  "dvp-api == 1.7.0.dev0",
   "dvp-common == {}".format(version),
   "enum34;python_version < '3.4'",
 ]
diff --git a/platform/src/main/python/dlpx/virtualization/platform/VERSION b/platform/src/main/python/dlpx/virtualization/platform/VERSION
@@ -1 +1 @@
-4.0.4
+4.0.5.dev1
diff --git a/platform/src/main/python/dlpx/virtualization/platform/_linked.py b/platform/src/main/python/dlpx/virtualization/platform/_linked.py
@@ -32,6 +32,7 @@ def __init__(self):
         self.status_impl = None
         self.worker_impl = None
         self.mount_specification_impl = None
+        self.source_size_impl = None
 
     def pre_snapshot(self):
         def pre_snapshot_decorator(pre_snapshot_impl):
@@ -101,6 +102,16 @@ def mount_specification_decorator(mount_specification_impl):
 
         return mount_specification_decorator
 
+    def source_size(self):
+        def source_size_decorator(source_size_impl):
+            if self.source_size_impl:
+                raise OperationAlreadyDefinedError(Op.LINKED_SOURCE_SIZE)
+            self.source_size_impl = v.check_function(source_size_impl,
+                                                     Op.LINKED_SOURCE_SIZE)
+            return source_size_impl
+
+        return source_size_decorator
+
     def _internal_direct_pre_snapshot(self, request):
         """Pre Snapshot Wrapper for direct plugins.
 
@@ -237,6 +248,57 @@ def to_protobuf(snapshot):
 
         return direct_post_snapshot_response
 
+    def _internal_direct_source_size(self, request):
+        """Direct Source Size Wrapper for direct plugins.
+
+        Executed as part of several operations to get the source size
+        of a direct source
+
+        Run source_size operation for a direct source.
+
+        Args:
+           request (DirectSourceSizeRequest): Source Size Request arguments.
+
+        Returns:
+           DirectSourceSizeResponse: A response containing the return value -
+           DirectSourceSizeResult which has source size. In
+           case of errors, response object will contain PluginErrorResult.
+        """
+        # Reasoning for method imports are in this file's docstring.
+        from generated.definitions import RepositoryDefinition
+        from generated.definitions import LinkedSourceDefinition
+        from generated.definitions import SourceConfigDefinition
+
+        #
+        # While linked.source_size() is not a required operation, this should
+        # not be called if it wasn't implemented.
+        #
+        if not self.source_size_impl:
+            raise OperationNotDefinedError(Op.LINKED_SOURCE_SIZE)
+
+        direct_source_definition = LinkedSourceDefinition.from_dict(
+            json.loads(request.direct_source.linked_source.parameters.json))
+        direct_source = DirectSource(
+            guid=request.direct_source.linked_source.guid,
+            connection=RemoteConnection.from_proto(
+                request.direct_source.connection),
+            parameters=direct_source_definition)
+
+        repository = RepositoryDefinition.from_dict(
+            json.loads(request.repository.parameters.json))
+        source_config = SourceConfigDefinition.from_dict(
+            json.loads(request.source_config.parameters.json))
+
+        source_size = self.source_size_impl(
+            direct_source=direct_source,
+            repository=repository,
+            source_config=source_config)
+
+        direct_source_size_response = platform_pb2.DirectSourceSizeResponse()
+        direct_source_size_response.return_value.database_size = source_size
+
+        return direct_source_size_response
+
     def _internal_staged_pre_snapshot(self, request):
         """Pre Snapshot Wrapper for staged plugins.
 
@@ -272,8 +334,8 @@ def _internal_staged_pre_snapshot(self, request):
         staged_mount = request.staged_source.staged_mount
         mount = Mount(remote_environment=RemoteEnvironment.from_proto(
             staged_mount.remote_environment),
-                      mount_path=staged_mount.mount_path,
-                      shared_path=staged_mount.shared_path)
+            mount_path=staged_mount.mount_path,
+            shared_path=staged_mount.shared_path)
         staged_source = StagedSource(
             guid=linked_source.guid,
             source_connection=RemoteConnection.from_proto(
@@ -719,3 +781,62 @@ def to_protobuf_ownership_spec(ownership_spec):
                 ownership_spec)
 
         return staged_mount_spec_response
+
+    def _internal_staged_source_size(self, request):
+        """Staged Source Size Wrapper for staged plugins.
+
+        Executed as part of several operations to get the source size
+        of a staged source
+
+        Run source_size operation for a staged source.
+
+        Args:
+           request (StagedSourceSizeRequest): Source Size Request arguments.
+
+        Returns:
+           StagedSourceSizeResponse: A response containing the return value -
+           StagedSourceSizeResult which has source size. In
+           case of errors, response object will contain PluginErrorResult.
+        """
+        # Reasoning for method imports are in this file's docstring.
+        from generated.definitions import RepositoryDefinition
+        from generated.definitions import LinkedSourceDefinition
+        from generated.definitions import SourceConfigDefinition
+
+        #
+        # While linked.source_size() is not a required operation, this should
+        # not be called if it wasn't implemented.
+        #
+        if not self.source_size_impl:
+            raise OperationNotDefinedError(Op.LINKED_SOURCE_SIZE)
+
+        staged_source_definition = LinkedSourceDefinition.from_dict(
+            json.loads(request.staged_source.linked_source.parameters.json))
+        mount = Mount(
+            remote_environment=(RemoteEnvironment.from_proto(
+                request.staged_source.staged_mount.remote_environment)),
+            mount_path=request.staged_source.staged_mount.mount_path,
+            shared_path=request.staged_source.staged_mount.shared_path)
+        staged_source = StagedSource(
+            guid=request.staged_source.linked_source.guid,
+            source_connection=RemoteConnection.from_proto(
+                request.staged_source.source_connection),
+            parameters=staged_source_definition,
+            mount=mount,
+            staged_connection=RemoteConnection.from_proto(
+                request.staged_source.staged_connection))
+
+        repository = RepositoryDefinition.from_dict(
+            json.loads(request.repository.parameters.json))
+        source_config = SourceConfigDefinition.from_dict(
+            json.loads(request.source_config.parameters.json))
+
+        source_size = self.source_size_impl(
+            staged_source=staged_source,
+            repository=repository,
+            source_config=source_config)
+
+        staged_source_size_response = platform_pb2.StagedSourceSizeResponse()
+        staged_source_size_response.return_value.database_size = source_size
+
+        return staged_source_size_response
diff --git a/platform/src/main/python/dlpx/virtualization/platform/_virtual.py b/platform/src/main/python/dlpx/virtualization/platform/_virtual.py
@@ -34,6 +34,7 @@ def __init__(self):
         self.status_impl = None
         self.initialize_impl = None
         self.mount_specification_impl = None
+        self.source_size_impl = None
 
     def configure(self):
         def configure_decorator(configure_impl):
@@ -142,6 +143,16 @@ def mount_specification_decorator(mount_specification_impl):
 
         return mount_specification_decorator
 
+    def source_size(self):
+        def source_size_decorator(source_size_impl):
+            if self.source_size_impl:
+                raise OperationAlreadyDefinedError(Op.VIRTUAL_SOURCE_SIZE)
+            self.source_size_impl = v.check_function(source_size_impl,
+                                                     Op.VIRTUAL_SOURCE_SIZE)
+            return source_size_impl
+
+        return source_size_decorator
+
     @staticmethod
     def _from_protobuf_single_subset_mount(single_subset_mount):
         return Mount(remote_environment=RemoteEnvironment.from_proto(
@@ -783,3 +794,57 @@ def to_protobuf_ownership_spec(ownership_spec):
         ]
         virtual_mount_spec_response.return_value.mounts.extend(mounts_list)
         return virtual_mount_spec_response
+
+    def _internal_virtual_source_size(self, request):
+        """Virtual Source Size Wrapper.
+
+        Executed as part of several operations to get the virtual source size
+        of a virtual source
+
+        Run source_size operation for a virtual source.
+
+        Args:
+           request (VirtualSourceSizeRequest): Source Size Request arguments.
+
+        Returns:
+           VirtualSourceSizeResponse: A response containing the return value -
+           VirtualSourceSizeResult which has source size. In
+           case of errors, response object will contain PluginErrorResult.
+        """
+        # Reasoning for method imports are in this file's docstring.
+        from generated.definitions import VirtualSourceDefinition
+        from generated.definitions import RepositoryDefinition
+        from generated.definitions import SourceConfigDefinition
+
+        #
+        # While virtual.source_size() is not a required operation, this should
+        # not be called if it wasn't implemented.
+        #
+        if not self.source_size_impl:
+            raise OperationNotDefinedError(Op.VIRTUAL_SOURCE_SIZE)
+
+        virtual_source_definition = VirtualSourceDefinition.from_dict(
+            json.loads(request.virtual_source.parameters.json))
+        mounts = [
+            VirtualOperations._from_protobuf_single_subset_mount(m)
+            for m in request.virtual_source.mounts
+        ]
+        virtual_source = VirtualSource(guid=request.virtual_source.guid,
+                                       connection=RemoteConnection.from_proto(
+                                           request.virtual_source.connection),
+                                       parameters=virtual_source_definition,
+                                       mounts=mounts)
+        repository = RepositoryDefinition.from_dict(
+            json.loads(request.repository.parameters.json))
+        source_config = SourceConfigDefinition.from_dict(
+            json.loads(request.source_config.parameters.json))
+
+        source_size = self.source_size_impl(
+            virtual_source=virtual_source,
+            repository=repository,
+            source_config=source_config)
+
+        virtual_source_size_response = platform_pb2.VirtualSourceSizeResponse()
+        virtual_source_size_response.return_value.database_size = source_size
+
+        return virtual_source_size_response
diff --git a/platform/src/main/python/dlpx/virtualization/platform/operation.py b/platform/src/main/python/dlpx/virtualization/platform/operation.py
@@ -17,6 +17,7 @@ class Operation(Enum):
     LINKED_STATUS = 'linked.status()'
     LINKED_WORKER = 'linked.worker()'
     LINKED_MOUNT_SPEC = 'linked.mount_specification()'
+    LINKED_SOURCE_SIZE = 'linked.source_size()'
 
     VIRTUAL_CONFIGURE = 'virtual.configure()'
     VIRTUAL_UNCONFIGURE = 'virtual.unconfigure()'
@@ -29,6 +30,7 @@ class Operation(Enum):
     VIRTUAL_STATUS = 'virtual.status()'
     VIRTUAL_INITIALIZE = 'virtual.initialize()'
     VIRTUAL_MOUNT_SPEC = 'virtual.mount_specification()'
+    VIRTUAL_SOURCE_SIZE = 'virtual.source_size()'
 
     UPGRADE_REPOSITORY = 'upgrade.repository()'
     UPGRADE_SOURCE_CONFIG = 'upgrade.source_config()'
diff --git a/platform/src/test/python/dlpx/virtualization/test_plugin.py b/platform/src/test/python/dlpx/virtualization/test_plugin.py
@@ -65,8 +65,8 @@
 }))
 TEST_POST_UPGRADE_PARAMS = ({
     u'obj':
-    '"{\\"obj\\": {\\"name\\": \\"upgrade\\", '
-    '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"'
+        '"{\\"obj\\": {\\"name\\": \\"upgrade\\", '
+        '\\"prettyName\\": \\"prettyUpgrade\\", \\"metadata\\": \\"metadata\\"}}"'
 })
 
 MIGRATION_IDS = ('2020.1.1', '2020.2.2')
@@ -94,7 +94,6 @@ def configure_impl():
             pass
 
         with pytest.raises(OperationAlreadyDefinedError):
-
             @my_plugin.virtual.configure()  # noqa F811
             def configure_impl():  # noqa F811
                 pass
@@ -1189,6 +1188,26 @@ def staged_status_impl(staged_source, repository, source_config):
 
         assert staged_status_response.return_value.status == expected_status
 
+    @staticmethod
+    def test_source_size(my_plugin, staged_source, repository,
+                         source_config):
+
+        @my_plugin.linked.source_size()
+        def staged_source_size_impl(staged_source, repository, source_config):
+            TestPlugin.assert_plugin_args(staged_source=staged_source,
+                                          repository=repository,
+                                          source_config=source_config)
+            return 0
+        staged_source_size_request = platform_pb2.StagedSourceSizeRequest()
+        TestPlugin.setup_request(request=staged_source_size_request,
+                                 staged_source=staged_source,
+                                 repository=repository,
+                                 source_config=source_config)
+        staged_source_size_response = my_plugin.linked._internal_staged_source_size(
+            staged_source_size_request)
+
+        assert staged_source_size_response.return_value.database_size == 0
+
     @staticmethod
     def test_staged_worker(my_plugin, staged_source, repository,
                            source_config):
diff --git a/tools/src/main/python/dlpx/virtualization/_internal/VERSION b/tools/src/main/python/dlpx/virtualization/_internal/VERSION
@@ -1 +1 @@
-4.0.4
+4.0.5.dev1
diff --git a/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py b/tools/src/main/python/dlpx/virtualization/_internal/plugin_importer.py
diff --git a/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg b/tools/src/main/python/dlpx/virtualization/_internal/settings.cfg
diff --git a/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml b/tools/src/main/python/dlpx/virtualization/_internal/validation_schemas/plugin_importer.yaml
diff --git a/tools/src/test/python/dlpx/virtualization/_internal/conftest.py b/tools/src/test/python/dlpx/virtualization/_internal/conftest.py
diff --git a/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py b/tools/src/test/python/dlpx/virtualization/_internal/fake_plugin/direct/successful.py
diff --git a/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py b/tools/src/test/python/dlpx/virtualization/_internal/test_package_util.py

Original file line number	Diff line number	Diff line change
`@@ -4,7 +4,7 @@`
`4`	`4`	`PYTHON_SRC = 'src/main/python'`
`5`	`5`
`6`	`6`	`install_requires = [`
`7`		`- "dvp-api == 1.6.3",`
	`7`	`+ "dvp-api == 1.7.0.dev0",`
`8`	`8`	`]`
`9`	`9`
`10`	`10`	`with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file:`
Original file line number	Diff line number	Diff line change
`@@ -7,7 +7,7 @@`
`7`	`7`	`version = version_file.read().strip()`
`8`	`8`
`9`	`9`	`install_requires = [`
`10`		`- "dvp-api == 1.6.3",`
	`10`	`+ "dvp-api == 1.7.0.dev0",`
`11`	`11`	`"dvp-common == {}".format(version)`
`12`	`12`	`]`
`13`	`13`